2023-05-06 23:53:00,647 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-06 23:53:00,933 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-06 23:53:01,034 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-06 23:53:01,092 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-06 23:53:01,099 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-06 23:53:01,111 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=339, ProcessCount=174, AvailableMemoryMB=4484 2023-05-06 23:53:01,113 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-06 23:53:01,113 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-06 23:53:01,250 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009518921:84 , srv1038722163:84 , srv1068344344:84 , srv1185107782:84 , srv1217337548:84 , srv124059105:84 , srv1261922907:84 , srv1280829067:84 , srv13055827:84 , srv1311393387:84 , srv1329834784:84 , srv1360927374:84 , srv1361737377:84 , srv1378027254:84 , srv1382723527:84 , srv1453470721:84 , srv1482070083:84 , srv1499624961:84 , srv1516374359:84 , srv1537526259:84 , srv1544875732:84 , srv1554963613:84 , srv1560074017:84 , srv1567758103:84 , srv1646979465:84 , srv1657787758:84 , srv1661434727:84 , srv1673599192:84 , srv1702152278:84 , srv1712174382:84 , srv1715613053:84 , srv1720331058:84 , srv17266886:84 , srv1739056313:84 , srv1748360339:84 , srv1765520665:84 , srv1782028788:84 , srv1784765561:84 , srv1795870300:84 , srv1831649703:84 , srv1833478769:84 , srv1841746872:84 , srv1843641730:84 , srv1846140482:84 , srv1860890916:84 , srv187853949:84 , srv188663595:84 , srv189254369:84 , srv1930871831:84 , srv1950744706:84 , srv1974133959:84 , srv1983555394:84 , srv2010308739:84 , srv2033575520:84 , srv2054098067:84 , srv2060991360:84 , srv2095019885:84 , srv2099489340:84 , srv2116553526:84 , srv2137949008:84 , srv259581389:84 , srv281842507:84 , srv285283594:84 , srv294421279:84 , srv298089849:84 , srv308500202:84 , srv339484107:84 , srv387200021:84 , srv395852870:84 , srv409837809:84 , srv434906644:84 , srv440767518:84 , srv456845511:84 , srv468910239:84 , srv471209957:84 , srv559192394:84 , srv576512274:84 , srv585373677:84 , srv604715756:84 , srv612810279:84 , srv647369764:84 , srv648461289:84 , srv656816565:84 , srv689453752:84 , srv72527762:84 , srv726446325:84 , srv733579217:84 , srv790457377:84 , srv790786340:84 , srv804073686:84 , srv813249878:84 , srv815415641:84 , srv852970707:84 , srv886927742:84 , srv892391183:84 , srv925854774:84 , srv93407435:84 , srv955491233:84 , srv990119240:84 , srv3659238:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-06 23:53:01,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:01,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2054098067=54, srv1537526259=19, srv1646979465=24, srv1784765561=37, srv612810279=80, srv852970707=93, srv1361737377=12, srv1217337548=4, srv468910239=74, srv1544875732=20, srv990119240=99, srv1930871831=48, srv308500202=65, srv187853949=45, srv892391183=95, srv804073686=90, srv1748360339=34, srv1782028788=36, srv124059105=5, srv1833478769=40, srv1360927374=11, srv1843641730=42, srv1846140482=43, srv440767518=72, srv1720331058=31, srv1554963613=21, srv2010308739=52, srv656816565=83, srv2116553526=58, srv604715756=79, srv689453752=84, srv1739056313=33, srv1329834784=10, srv733579217=87, srv294421279=63, srv559192394=76, srv281842507=61, srv1712174382=29, srv93407435=97, srv1974133959=50, srv2095019885=56, srv1261922907=6, srv1567758103=23, srv1499624961=17, srv2033575520=53, srv434906644=71, srv298089849=64, srv13055827=8, srv1657787758=25, srv2060991360=55, srv647369764=81, srv1841746872=41, srv471209957=75, srv1038722163=1, srv456845511=73, srv576512274=77, srv1378027254=13, srv1795870300=38, srv1068344344=2, srv1950744706=49, srv1765520665=35, srv1382723527=14, srv1453470721=15, srv1715613053=30, srv925854774=96, srv1983555394=51, srv188663595=46, srv2137949008=59, srv1311393387=9, srv387200021=68, srv1482070083=16, srv1673599192=27, srv1009518921=0, srv1280829067=7, srv1661434727=26, srv2099489340=57, srv1702152278=28, srv17266886=32, srv189254369=47, srv409837809=70, srv285283594=62, srv3659238=67, srv648461289=82, srv395852870=69, srv72527762=85, srv726446325=86, srv790786340=89, srv1860890916=44, srv1185107782=3, srv259581389=60, srv815415641=92, srv339484107=66, srv1831649703=39, srv1516374359=18, srv1560074017=22, srv955491233=98, srv790457377=88, srv813249878=91, srv585373677=78, srv886927742=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:01,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:01,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:01,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:01,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:01,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:01,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:01,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:01,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:01,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:01,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:01,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:01,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:01,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-06 23:53:01,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-06 23:53:01,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-06 23:53:01,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-06 23:53:01,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-06 23:53:01,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-06 23:53:01,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-06 23:53:01,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-06 23:53:01,649 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:01,649 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-06 23:53:01,649 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-06 23:53:02,096 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 724 ms to try 5000 different iterations. Found a solution that moves 4690 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1819243699097267. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04152564988598821, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05683180532519413, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04152564988598821, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:02,105 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1657787758:82 , srv339484107:82 , srv409837809:82 , srv815415641:82 , srv124059105:83 , srv1280829067:83 , srv1782028788:83 , srv188663595:83 , srv1974133959:83 , srv434906644:83 , srv468910239:83 , srv790786340:83 , srv804073686:83 , srv886927742:83 , srv1261922907:84 , srv1311393387:84 , srv1360927374:84 , srv1382723527:84 , srv1482070083:84 , srv1499624961:84 , srv1748360339:84 , srv2054098067:84 , srv2137949008:84 , srv285283594:84 , srv294421279:84 , srv395852870:84 , srv852970707:84 , srv990119240:84 , srv1009518921:85 , srv1661434727:85 , srv1673599192:85 , srv1720331058:85 , srv1795870300:85 , srv187853949:85 , srv189254369:85 , srv259581389:85 , srv585373677:85 , srv656816565:85 , srv689453752:85 , srv72527762:85 , srv726446325:85 , srv790457377:85 , srv813249878:85 , srv892391183:85 , srv955491233:85 , srv1217337548:86 , srv1453470721:86 , srv1554963613:86 , srv1702152278:86 , srv1715613053:86 , srv1765520665:86 , srv1930871831:86 , srv2033575520:86 , srv2060991360:86 , srv281842507:86 , srv604715756:86 , srv648461289:86 , srv925854774:86 , srv1516374359:87 , srv17266886:87 , srv1833478769:87 , srv1860890916:87 , srv2010308739:87 , srv456845511:87 , srv471209957:87 , srv559192394:87 , srv576512274:87 , srv612810279:87 , srv93407435:87 , srv1068344344:88 , srv13055827:88 , srv1537526259:88 , srv1567758103:88 , srv1646979465:88 , srv1712174382:88 , srv1841746872:88 , srv2099489340:88 , srv2116553526:88 , srv308500202:88 , srv440767518:88 , srv1038722163:89 , srv1185107782:89 , srv1378027254:89 , srv1544875732:89 , srv1843641730:89 , srv1950744706:89 , srv1983555394:89 , srv2095019885:89 , srv647369764:89 , srv733579217:89 , srv1329834784:90 , srv387200021:90 , srv1361737377:91 , srv1560074017:91 , srv1739056313:91 , srv298089849:92 , srv1846140482:93 , srv1784765561:94 , srv1831649703:97 , srv3659238:459 } 2023-05-06 23:53:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2054098067=54, srv1537526259=19, srv1646979465=24, srv1784765561=37, srv612810279=80, srv852970707=93, srv1361737377=12, srv1217337548=4, srv468910239=74, srv1544875732=20, srv990119240=99, srv1930871831=48, srv308500202=65, srv187853949=45, srv892391183=95, srv804073686=90, srv1748360339=34, srv1782028788=36, srv124059105=5, srv1833478769=40, srv1360927374=11, srv1843641730=42, srv1846140482=43, srv440767518=72, srv1720331058=31, srv1554963613=21, srv2010308739=52, srv656816565=83, srv2116553526=58, srv604715756=79, srv689453752=84, srv1739056313=33, srv1329834784=10, srv733579217=87, srv294421279=63, srv559192394=76, srv281842507=61, srv1712174382=29, srv93407435=97, srv1974133959=50, srv2095019885=56, srv1261922907=6, srv1567758103=23, srv1499624961=17, srv2033575520=53, srv434906644=71, srv298089849=64, srv13055827=8, srv1657787758=25, srv2060991360=55, srv647369764=81, srv1841746872=41, srv471209957=75, srv1038722163=1, srv456845511=73, srv576512274=77, srv1378027254=13, srv1795870300=38, srv1068344344=2, srv1950744706=49, srv1765520665=35, srv1382723527=14, srv1453470721=15, srv1715613053=30, srv925854774=96, srv1983555394=51, srv188663595=46, srv2137949008=59, srv1311393387=9, srv387200021=68, srv1482070083=16, srv1673599192=27, srv1009518921=0, srv1280829067=7, srv1661434727=26, srv2099489340=57, srv1702152278=28, srv17266886=32, srv189254369=47, srv409837809=70, srv285283594=62, srv3659238=67, srv648461289=82, srv395852870=69, srv72527762=85, srv726446325=86, srv790786340=89, srv1860890916=44, srv1185107782=3, srv259581389=60, srv815415641=92, srv339484107=66, srv1831649703=39, srv1516374359=18, srv1560074017=22, srv955491233=98, srv790457377=88, srv813249878=91, srv585373677=78, srv886927742=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-06 23:53:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-06 23:53:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-06 23:53:02,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-06 23:53:02,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-06 23:53:02,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-06 23:53:02,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-06 23:53:02,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-06 23:53:02,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-06 23:53:02,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04152564988598821, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05683180532519413, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04152564988598821, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:02,261 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-06 23:53:02,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18189152982782156, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04152564988598821, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05683180532519413, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04152564988598821, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-06 23:53:02,463 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 292 ms to try 5000 different iterations. Found a solution that moves 2688 regions; Going from a computed imbalance of 0.18189152982782156 to a new imbalance of 0.014694740368570627. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02316517892016576); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03719394372819474, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2986666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02316517892016576); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:02,472 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1831649703:75 , srv1846140482:78 , srv1784765561:79 , srv387200021:80 , srv1361737377:81 , srv1378027254:81 , srv1560074017:81 , srv2095019885:81 , srv1739056313:82 , srv1841746872:82 , srv1950744706:82 , srv1983555394:82 , srv1038722163:83 , srv1185107782:83 , srv1329834784:83 , srv1843641730:83 , srv2116553526:83 , srv298089849:83 , srv647369764:83 , srv93407435:83 , srv1068344344:84 , srv13055827:84 , srv1453470721:84 , srv1516374359:84 , srv1537526259:84 , srv1554963613:84 , srv1646979465:84 , srv308500202:84 , srv471209957:84 , srv733579217:84 , srv1544875732:85 , srv1567758103:85 , srv1712174382:85 , srv17266886:85 , srv1833478769:85 , srv1930871831:85 , srv1974133959:85 , srv2010308739:85 , srv2033575520:85 , srv2099489340:85 , srv440767518:85 , srv559192394:85 , srv585373677:85 , srv604715756:85 , srv648461289:85 , srv1009518921:86 , srv1382723527:86 , srv1673599192:86 , srv1702152278:86 , srv1715613053:86 , srv1860890916:86 , srv2054098067:86 , srv2060991360:86 , srv285283594:86 , srv456845511:86 , srv576512274:86 , srv689453752:86 , srv813249878:86 , srv1482070083:87 , srv1661434727:87 , srv1720331058:87 , srv1765520665:87 , srv1795870300:87 , srv2137949008:87 , srv259581389:87 , srv294421279:87 , srv726446325:87 , srv790457377:87 , srv804073686:87 , srv852970707:87 , srv925854774:87 , srv955491233:87 , srv1217337548:88 , srv124059105:88 , srv1261922907:88 , srv1311393387:88 , srv1748360339:88 , srv187853949:88 , srv188663595:88 , srv189254369:88 , srv434906644:88 , srv612810279:88 , srv656816565:88 , srv886927742:88 , srv892391183:88 , srv990119240:88 , srv1360927374:89 , srv1782028788:89 , srv395852870:89 , srv409837809:89 , srv790786340:89 , srv1280829067:90 , srv1499624961:90 , srv281842507:90 , srv72527762:90 , srv815415641:90 , srv1657787758:91 , srv339484107:91 , srv468910239:91 , srv3659238:521 } 2023-05-06 23:53:02,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:02,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2054098067=54, srv1537526259=19, srv1646979465=24, srv1784765561=37, srv612810279=80, srv852970707=93, srv1361737377=12, srv1217337548=4, srv468910239=74, srv1544875732=20, srv990119240=99, srv1930871831=48, srv308500202=65, srv187853949=45, srv892391183=95, srv804073686=90, srv1748360339=34, srv1782028788=36, srv124059105=5, srv1833478769=40, srv1360927374=11, srv1843641730=42, srv1846140482=43, srv440767518=72, srv1720331058=31, srv1554963613=21, srv2010308739=52, srv656816565=83, srv2116553526=58, srv604715756=79, srv689453752=84, srv1739056313=33, srv1329834784=10, srv733579217=87, srv294421279=63, srv559192394=76, srv281842507=61, srv1712174382=29, srv93407435=97, srv1974133959=50, srv2095019885=56, srv1261922907=6, srv1567758103=23, srv1499624961=17, srv2033575520=53, srv434906644=71, srv298089849=64, srv13055827=8, srv1657787758=25, srv2060991360=55, srv647369764=81, srv1841746872=41, srv471209957=75, srv1038722163=1, srv456845511=73, srv576512274=77, srv1378027254=13, srv1795870300=38, srv1068344344=2, srv1950744706=49, srv1765520665=35, srv1382723527=14, srv1453470721=15, srv1715613053=30, srv925854774=96, srv1983555394=51, srv188663595=46, srv2137949008=59, srv1311393387=9, srv387200021=68, srv1482070083=16, srv1673599192=27, srv1009518921=0, srv1280829067=7, srv1661434727=26, srv2099489340=57, srv1702152278=28, srv17266886=32, srv189254369=47, srv409837809=70, srv285283594=62, srv3659238=67, srv648461289=82, srv395852870=69, srv72527762=85, srv726446325=86, srv790786340=89, srv1860890916=44, srv1185107782=3, srv259581389=60, srv815415641=92, srv339484107=66, srv1831649703=39, srv1516374359=18, srv1560074017=22, srv955491233=98, srv790457377=88, srv813249878=91, srv585373677=78, srv886927742=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:02,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:02,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:02,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:02,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:02,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:02,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-06 23:53:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-06 23:53:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-06 23:53:02,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-06 23:53:02,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-06 23:53:02,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-06 23:53:02,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-06 23:53:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-06 23:53:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-06 23:53:02,598 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02316517892016576); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03719394372819474, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02316517892016576); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:02,598 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-06 23:53:02,598 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014675918590284692, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02316517892016576); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03719394372819474, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02316517892016576); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-06 23:53:02,765 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 243 ms to try 5000 different iterations. Found a solution that moves 1029 regions; Going from a computed imbalance of 0.014675918590284692 to a new imbalance of 7.874194780340111E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005329871678683814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007624757360309341); MoveCostFunction : (multiplier=7.0, imbalance=0.11433333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005329871678683814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:02,766 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv281842507:82 , srv468910239:82 , srv72527762:82 , srv1499624961:83 , srv298089849:83 , srv1185107782:84 , srv1217337548:84 , srv1329834784:84 , srv1360927374:84 , srv1544875732:84 , srv1567758103:84 , srv1712174382:84 , srv1739056313:84 , srv1765520665:84 , srv1784765561:84 , srv1831649703:84 , srv187853949:84 , srv2099489340:84 , srv395852870:84 , srv612810279:84 , srv1280829067:85 , srv1311393387:85 , srv1537526259:85 , srv1560074017:85 , srv1657787758:85 , srv1673599192:85 , srv17266886:85 , srv1833478769:85 , srv1843641730:85 , srv1860890916:85 , srv189254369:85 , srv2060991360:85 , srv339484107:85 , srv440767518:85 , srv456845511:85 , srv656816565:85 , srv733579217:85 , srv790786340:85 , srv815415641:85 , srv892391183:85 , srv925854774:85 , srv955491233:85 , srv990119240:85 , srv1038722163:86 , srv1068344344:86 , srv124059105:86 , srv1261922907:86 , srv13055827:86 , srv1361737377:86 , srv1482070083:86 , srv1516374359:86 , srv1646979465:86 , srv1661434727:86 , srv1702152278:86 , srv1715613053:86 , srv1720331058:86 , srv1782028788:86 , srv1795870300:86 , srv1930871831:86 , srv1983555394:86 , srv2010308739:86 , srv2033575520:86 , srv2137949008:86 , srv294421279:86 , srv308500202:86 , srv409837809:86 , srv559192394:86 , srv576512274:86 , srv585373677:86 , srv604715756:86 , srv647369764:86 , srv689453752:86 , srv726446325:86 , srv790457377:86 , srv813249878:86 , srv852970707:86 , srv1378027254:87 , srv1382723527:87 , srv1748360339:87 , srv1841746872:87 , srv1846140482:87 , srv188663595:87 , srv1950744706:87 , srv2054098067:87 , srv2095019885:87 , srv2116553526:87 , srv259581389:87 , srv434906644:87 , srv471209957:87 , srv648461289:87 , srv804073686:87 , srv886927742:87 , srv93407435:87 , srv1009518921:88 , srv1453470721:88 , srv1554963613:88 , srv387200021:88 , srv285283594:89 , srv1974133959:90 , srv3659238:525 } 2023-05-06 23:53:02,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:02,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2054098067=54, srv1537526259=19, srv1646979465=24, srv1784765561=37, srv612810279=80, srv852970707=93, srv1361737377=12, srv1217337548=4, srv468910239=74, srv1544875732=20, srv990119240=99, srv1930871831=48, srv308500202=65, srv187853949=45, srv892391183=95, srv804073686=90, srv1748360339=34, srv1782028788=36, srv124059105=5, srv1833478769=40, srv1360927374=11, srv1843641730=42, srv1846140482=43, srv440767518=72, srv1720331058=31, srv1554963613=21, srv2010308739=52, srv656816565=83, srv2116553526=58, srv604715756=79, srv689453752=84, srv1739056313=33, srv1329834784=10, srv733579217=87, srv294421279=63, srv559192394=76, srv281842507=61, srv1712174382=29, srv93407435=97, srv1974133959=50, srv2095019885=56, srv1261922907=6, srv1567758103=23, srv1499624961=17, srv2033575520=53, srv434906644=71, srv298089849=64, srv13055827=8, srv1657787758=25, srv2060991360=55, srv647369764=81, srv1841746872=41, srv471209957=75, srv1038722163=1, srv456845511=73, srv576512274=77, srv1378027254=13, srv1795870300=38, srv1068344344=2, srv1950744706=49, srv1765520665=35, srv1382723527=14, srv1453470721=15, srv1715613053=30, srv925854774=96, srv1983555394=51, srv188663595=46, srv2137949008=59, srv1311393387=9, srv387200021=68, srv1482070083=16, srv1673599192=27, srv1009518921=0, srv1280829067=7, srv1661434727=26, srv2099489340=57, srv1702152278=28, srv17266886=32, srv189254369=47, srv409837809=70, srv285283594=62, srv3659238=67, srv648461289=82, srv395852870=69, srv72527762=85, srv726446325=86, srv790786340=89, srv1860890916=44, srv1185107782=3, srv259581389=60, srv815415641=92, srv339484107=66, srv1831649703=39, srv1516374359=18, srv1560074017=22, srv955491233=98, srv790457377=88, srv813249878=91, srv585373677=78, srv886927742=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:02,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:02,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:02,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:02,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:02,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-06 23:53:02,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-06 23:53:02,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-06 23:53:02,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-06 23:53:02,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-06 23:53:02,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-06 23:53:02,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-06 23:53:02,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-06 23:53:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-06 23:53:02,867 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005329871678683814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007624757360309341); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005329871678683814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:02,867 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-06 23:53:02,867 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.802142660339272E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005329871678683814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007624757360309341); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005329871678683814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-06 23:53:03,078 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 282 ms to try 5000 different iterations. Found a solution that moves 284 regions; Going from a computed imbalance of 7.802142660339272E-4 to a new imbalance of 3.543155231690461E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.03155555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:03,079 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv285283594:82 , srv1009518921:83 , srv1554963613:83 , srv1748360339:83 , srv1950744706:83 , srv1974133959:83 , srv259581389:83 , srv576512274:83 , srv612810279:83 , srv1038722163:84 , srv1068344344:84 , srv1217337548:84 , srv1261922907:84 , srv1280829067:84 , srv13055827:84 , srv1360927374:84 , srv1361737377:84 , srv1453470721:84 , srv1544875732:84 , srv1657787758:84 , srv1661434727:84 , srv1715613053:84 , srv1720331058:84 , srv1765520665:84 , srv1846140482:84 , srv1860890916:84 , srv188663595:84 , srv1983555394:84 , srv2010308739:84 , srv2116553526:84 , srv281842507:84 , srv298089849:84 , srv308500202:84 , srv339484107:84 , srv387200021:84 , srv409837809:84 , srv434906644:84 , srv440767518:84 , srv456845511:84 , srv471209957:84 , srv559192394:84 , srv647369764:84 , srv648461289:84 , srv656816565:84 , srv726446325:84 , srv733579217:84 , srv790457377:84 , srv813249878:84 , srv892391183:84 , srv925854774:84 , srv124059105:85 , srv1311393387:85 , srv1329834784:85 , srv1378027254:85 , srv1382723527:85 , srv1482070083:85 , srv1499624961:85 , srv1516374359:85 , srv1537526259:85 , srv1560074017:85 , srv1567758103:85 , srv1646979465:85 , srv1673599192:85 , srv1702152278:85 , srv17266886:85 , srv1739056313:85 , srv1782028788:85 , srv1795870300:85 , srv1841746872:85 , srv1843641730:85 , srv187853949:85 , srv189254369:85 , srv1930871831:85 , srv2033575520:85 , srv2054098067:85 , srv2060991360:85 , srv2095019885:85 , srv2137949008:85 , srv294421279:85 , srv395852870:85 , srv604715756:85 , srv689453752:85 , srv72527762:85 , srv790786340:85 , srv804073686:85 , srv815415641:85 , srv852970707:85 , srv886927742:85 , srv93407435:85 , srv955491233:85 , srv990119240:85 , srv1185107782:86 , srv1712174382:86 , srv1784765561:86 , srv1831649703:86 , srv1833478769:86 , srv2099489340:86 , srv468910239:86 , srv585373677:86 , srv3659238:637 } 2023-05-06 23:53:03,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:03,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2054098067=54, srv1537526259=19, srv1646979465=24, srv1784765561=37, srv612810279=80, srv852970707=93, srv1361737377=12, srv1217337548=4, srv468910239=74, srv1544875732=20, srv990119240=99, srv1930871831=48, srv308500202=65, srv187853949=45, srv892391183=95, srv804073686=90, srv1748360339=34, srv1782028788=36, srv124059105=5, srv1833478769=40, srv1360927374=11, srv1843641730=42, srv1846140482=43, srv440767518=72, srv1720331058=31, srv1554963613=21, srv2010308739=52, srv656816565=83, srv2116553526=58, srv604715756=79, srv689453752=84, srv1739056313=33, srv1329834784=10, srv733579217=87, srv294421279=63, srv559192394=76, srv281842507=61, srv1712174382=29, srv93407435=97, srv1974133959=50, srv2095019885=56, srv1261922907=6, srv1567758103=23, srv1499624961=17, srv2033575520=53, srv434906644=71, srv298089849=64, srv13055827=8, srv1657787758=25, srv2060991360=55, srv647369764=81, srv1841746872=41, srv471209957=75, srv1038722163=1, srv456845511=73, srv576512274=77, srv1378027254=13, srv1795870300=38, srv1068344344=2, srv1950744706=49, srv1765520665=35, srv1382723527=14, srv1453470721=15, srv1715613053=30, srv925854774=96, srv1983555394=51, srv188663595=46, srv2137949008=59, srv1311393387=9, srv387200021=68, srv1482070083=16, srv1673599192=27, srv1009518921=0, srv1280829067=7, srv1661434727=26, srv2099489340=57, srv1702152278=28, srv17266886=32, srv189254369=47, srv409837809=70, srv285283594=62, srv3659238=67, srv648461289=82, srv395852870=69, srv72527762=85, srv726446325=86, srv790786340=89, srv1860890916=44, srv1185107782=3, srv259581389=60, srv815415641=92, srv339484107=66, srv1831649703=39, srv1516374359=18, srv1560074017=22, srv955491233=98, srv790457377=88, srv813249878=91, srv585373677=78, srv886927742=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-06 23:53:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-06 23:53:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-06 23:53:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-06 23:53:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-06 23:53:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-06 23:53:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-06 23:53:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-06 23:53:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-06 23:53:03,171 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:03,171 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-06 23:53:03,171 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.344294181348005E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-06 23:53:03,313 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 209 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 3.344294181348005E-5 to a new imbalance of 6.940129244310425E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:03,313 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1646979465:83 , srv1702152278:83 , srv1712174382:83 , srv1782028788:83 , srv1784765561:83 , srv1795870300:83 , srv1833478769:83 , srv189254369:83 , srv2099489340:83 , srv886927742:83 , srv1009518921:84 , srv1038722163:84 , srv1068344344:84 , srv1185107782:84 , srv1217337548:84 , srv124059105:84 , srv1261922907:84 , srv1280829067:84 , srv13055827:84 , srv1311393387:84 , srv1329834784:84 , srv1361737377:84 , srv1378027254:84 , srv1382723527:84 , srv1453470721:84 , srv1482070083:84 , srv1499624961:84 , srv1516374359:84 , srv1537526259:84 , srv1544875732:84 , srv1560074017:84 , srv1567758103:84 , srv1657787758:84 , srv1661434727:84 , srv1715613053:84 , srv1720331058:84 , srv17266886:84 , srv1739056313:84 , srv1748360339:84 , srv1831649703:84 , srv1841746872:84 , srv1843641730:84 , srv1846140482:84 , srv1860890916:84 , srv187853949:84 , srv188663595:84 , srv1930871831:84 , srv2010308739:84 , srv2033575520:84 , srv2054098067:84 , srv2060991360:84 , srv2095019885:84 , srv2116553526:84 , srv2137949008:84 , srv259581389:84 , srv281842507:84 , srv294421279:84 , srv298089849:84 , srv308500202:84 , srv339484107:84 , srv387200021:84 , srv395852870:84 , srv434906644:84 , srv440767518:84 , srv456845511:84 , srv468910239:84 , srv471209957:84 , srv559192394:84 , srv576512274:84 , srv585373677:84 , srv604715756:84 , srv612810279:84 , srv647369764:84 , srv648461289:84 , srv656816565:84 , srv689453752:84 , srv72527762:84 , srv726446325:84 , srv733579217:84 , srv790457377:84 , srv790786340:84 , srv804073686:84 , srv815415641:84 , srv852970707:84 , srv892391183:84 , srv925854774:84 , srv93407435:84 , srv955491233:84 , srv990119240:84 , srv1360927374:85 , srv1554963613:85 , srv1673599192:85 , srv1765520665:85 , srv1950744706:85 , srv1974133959:85 , srv1983555394:85 , srv285283594:85 , srv409837809:85 , srv813249878:85 , srv3659238:684 } 2023-05-06 23:53:03,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:03,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2054098067=54, srv1537526259=19, srv1646979465=24, srv1784765561=37, srv612810279=80, srv852970707=93, srv1361737377=12, srv1217337548=4, srv468910239=74, srv1544875732=20, srv990119240=99, srv1930871831=48, srv308500202=65, srv187853949=45, srv892391183=95, srv804073686=90, srv1748360339=34, srv1782028788=36, srv124059105=5, srv1833478769=40, srv1360927374=11, srv1843641730=42, srv1846140482=43, srv440767518=72, srv1720331058=31, srv1554963613=21, srv2010308739=52, srv656816565=83, srv2116553526=58, srv604715756=79, srv689453752=84, srv1739056313=33, srv1329834784=10, srv733579217=87, srv294421279=63, srv559192394=76, srv281842507=61, srv1712174382=29, srv93407435=97, srv1974133959=50, srv2095019885=56, srv1261922907=6, srv1567758103=23, srv1499624961=17, srv2033575520=53, srv434906644=71, srv298089849=64, srv13055827=8, srv1657787758=25, srv2060991360=55, srv647369764=81, srv1841746872=41, srv471209957=75, srv1038722163=1, srv456845511=73, srv576512274=77, srv1378027254=13, srv1795870300=38, srv1068344344=2, srv1950744706=49, srv1765520665=35, srv1382723527=14, srv1453470721=15, srv1715613053=30, srv925854774=96, srv1983555394=51, srv188663595=46, srv2137949008=59, srv1311393387=9, srv387200021=68, srv1482070083=16, srv1673599192=27, srv1009518921=0, srv1280829067=7, srv1661434727=26, srv2099489340=57, srv1702152278=28, srv17266886=32, srv189254369=47, srv409837809=70, srv285283594=62, srv3659238=67, srv648461289=82, srv395852870=69, srv72527762=85, srv726446325=86, srv790786340=89, srv1860890916=44, srv1185107782=3, srv259581389=60, srv815415641=92, srv339484107=66, srv1831649703=39, srv1516374359=18, srv1560074017=22, srv955491233=98, srv790457377=88, srv813249878=91, srv585373677=78, srv886927742=94} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:03,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-06 23:53:03,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-06 23:53:03,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-06 23:53:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-06 23:53:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-06 23:53:03,403 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:03,404 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1646979465:83 , srv1702152278:83 , srv1712174382:83 , srv1782028788:83 , srv1784765561:83 , srv1795870300:83 , srv1833478769:83 , srv189254369:83 , srv2099489340:83 , srv886927742:83 , srv1009518921:84 , srv1038722163:84 , srv1068344344:84 , srv1185107782:84 , srv1217337548:84 , srv124059105:84 , srv1261922907:84 , srv1280829067:84 , srv13055827:84 , srv1311393387:84 , srv1329834784:84 , srv1361737377:84 , srv1378027254:84 , srv1382723527:84 , srv1453470721:84 , srv1482070083:84 , srv1499624961:84 , srv1516374359:84 , srv1537526259:84 , srv1544875732:84 , srv1560074017:84 , srv1567758103:84 , srv1657787758:84 , srv1661434727:84 , srv1715613053:84 , srv1720331058:84 , srv17266886:84 , srv1739056313:84 , srv1748360339:84 , srv1831649703:84 , srv1841746872:84 , srv1843641730:84 , srv1846140482:84 , srv1860890916:84 , srv187853949:84 , srv188663595:84 , srv1930871831:84 , srv2010308739:84 , srv2033575520:84 , srv2054098067:84 , srv2060991360:84 , srv2095019885:84 , srv2116553526:84 , srv2137949008:84 , srv259581389:84 , srv281842507:84 , srv294421279:84 , srv298089849:84 , srv308500202:84 , srv339484107:84 , srv387200021:84 , srv395852870:84 , srv434906644:84 , srv440767518:84 , srv456845511:84 , srv468910239:84 , srv471209957:84 , srv559192394:84 , srv576512274:84 , srv585373677:84 , srv604715756:84 , srv612810279:84 , srv647369764:84 , srv648461289:84 , srv656816565:84 , srv689453752:84 , srv72527762:84 , srv726446325:84 , srv733579217:84 , srv790457377:84 , srv790786340:84 , srv804073686:84 , srv815415641:84 , srv852970707:84 , srv892391183:84 , srv925854774:84 , srv93407435:84 , srv955491233:84 , srv990119240:84 , srv1360927374:85 , srv1554963613:85 , srv1673599192:85 , srv1765520665:85 , srv1950744706:85 , srv1974133959:85 , srv1983555394:85 , srv285283594:85 , srv409837809:85 , srv813249878:85 , srv3659238:684 } 2023-05-06 23:53:03,426 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=339 (was 339), ProcessCount=174 (was 174), AvailableMemoryMB=3869 (was 4484) 2023-05-06 23:53:03,433 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=339, ProcessCount=174, AvailableMemoryMB=3869 2023-05-06 23:53:03,434 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-06 23:53:03,434 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-06 23:53:03,441 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv104370883:84 , srv1369932073:84 , srv1391739299:84 , srv1433205713:84 , srv1577690916:84 , srv1635249814:84 , srv1675171426:84 , srv1814765411:84 , srv1853452248:84 , srv1867602407:84 , srv1897275658:84 , srv1978694885:84 , srv1999388630:84 , srv2141286696:84 , srv226587487:84 , srv239561997:84 , srv402411955:84 , srv446209643:84 , srv452538635:84 , srv456618950:84 , srv511354164:84 , srv51766974:84 , srv561262448:84 , srv652153001:84 , srv657811864:84 , srv661971122:84 , srv741389144:84 , srv864096991:84 , srv934169557:84 , srv1230583604:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-06 23:53:03,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:53:03,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv657811864=25, srv511354164=21, srv1978694885=12, srv51766974=22, srv1230583604=1, srv1391739299=3, srv456618950=20, srv239561997=16, srv226587487=15, srv652153001=24, srv1433205713=4, srv1675171426=7, srv1853452248=9, srv1867602407=10, srv741389144=27, srv864096991=28, srv934169557=29, srv1814765411=8, srv452538635=19, srv1369932073=2, srv1897275658=11, srv1635249814=6, srv2141286696=14, srv1999388630=13, srv661971122=26, srv104370883=0, srv446209643=18, srv1577690916=5, srv561262448=23, srv402411955=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:53:03,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:53:03,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-06 23:53:03,465 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1819939446392596, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:53:03,465 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912766500241187, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1819939446392596, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-06 23:55:03,454 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9544244 different iterations. Found a solution that moves 1842 regions; Going from a computed imbalance of 0.9912766500241187 to a new imbalance of 4.2993198912066E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:55:03,460 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv104370883:90 , srv1230583604:90 , srv1369932073:90 , srv1391739299:90 , srv1433205713:90 , srv1577690916:90 , srv1635249814:90 , srv1675171426:90 , srv1814765411:90 , srv1853452248:90 , srv1867602407:90 , srv1897275658:90 , srv1978694885:90 , srv1999388630:90 , srv2141286696:90 , srv226587487:90 , srv239561997:90 , srv402411955:90 , srv446209643:90 , srv452538635:90 , srv456618950:90 , srv511354164:90 , srv51766974:90 , srv561262448:90 , srv652153001:90 , srv657811864:90 , srv661971122:90 , srv741389144:90 , srv864096991:90 , srv934169557:90 } 2023-05-06 23:55:03,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-06 23:55:03,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv657811864=25, srv511354164=21, srv1978694885=12, srv51766974=22, srv1230583604=1, srv1391739299=3, srv456618950=20, srv239561997=16, srv226587487=15, srv652153001=24, srv1433205713=4, srv1675171426=7, srv1853452248=9, srv1867602407=10, srv741389144=27, srv864096991=28, srv934169557=29, srv1814765411=8, srv452538635=19, srv1369932073=2, srv1897275658=11, srv1635249814=6, srv2141286696=14, srv1999388630=13, srv661971122=26, srv104370883=0, srv446209643=18, srv1577690916=5, srv561262448=23, srv402411955=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-06 23:55:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-06 23:55:03,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-06 23:55:03,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-06 23:55:03,475 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv104370883:90 , srv1230583604:90 , srv1369932073:90 , srv1391739299:90 , srv1433205713:90 , srv1577690916:90 , srv1635249814:90 , srv1675171426:90 , srv1814765411:90 , srv1853452248:90 , srv1867602407:90 , srv1897275658:90 , srv1978694885:90 , srv1999388630:90 , srv2141286696:90 , srv226587487:90 , srv239561997:90 , srv402411955:90 , srv446209643:90 , srv452538635:90 , srv456618950:90 , srv511354164:90 , srv51766974:90 , srv561262448:90 , srv652153001:90 , srv657811864:90 , srv661971122:90 , srv741389144:90 , srv864096991:90 , srv934169557:90 } 2023-05-06 23:55:03,485 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=197 (was 339), ProcessCount=174 (was 174), AvailableMemoryMB=3122 (was 3869)