2024-03-24 13:53:27,799 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-03-24 13:53:28,081 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-03-24 13:53:28,189 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-03-24 13:53:28,250 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-03-24 13:53:28,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-03-24 13:53:28,272 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=170, AvailableMemoryMB=13967 2024-03-24 13:53:28,274 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-03-24 13:53:28,274 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-03-24 13:53:28,393 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008821124:84 , srv101698354:84 , srv1061407474:84 , srv1070227681:84 , srv1085820900:84 , srv1092967907:84 , srv1104483018:84 , srv113056904:84 , srv1160728679:84 , srv1180745137:84 , srv1190790200:84 , srv1201751666:84 , srv1217889997:84 , srv1218301266:84 , srv1249407175:84 , srv1281478803:84 , srv1285519461:84 , srv1300890373:84 , srv1324011916:84 , srv133637566:84 , srv1341984421:84 , srv1386103153:84 , srv1388622987:84 , srv1395599617:84 , srv1399881111:84 , srv1452227652:84 , srv14698537:84 , srv1470513626:84 , srv1488839012:84 , srv1506260242:84 , srv1536043520:84 , srv1553181010:84 , srv1554050261:84 , srv1560012132:84 , srv1584420490:84 , srv1620221588:84 , srv1625887396:84 , srv1635668467:84 , srv1643119471:84 , srv1644465393:84 , srv1646989268:84 , srv1649912388:84 , srv1659273052:84 , srv1675683664:84 , srv1688881717:84 , srv1726907993:84 , srv1749496064:84 , srv1780650489:84 , srv184793716:84 , srv1910885756:84 , srv1912798700:84 , srv1915571616:84 , srv1947001652:84 , srv1966279711:84 , srv1983573527:84 , srv198651878:84 , srv2004499467:84 , srv2009815200:84 , srv2025105443:84 , srv2027701415:84 , srv2039703914:84 , srv204462244:84 , srv2062164577:84 , srv2075732596:84 , srv2114629450:84 , srv2118644438:84 , srv235345887:84 , srv25056210:84 , srv270456563:84 , srv281106914:84 , srv291180855:84 , srv30665337:84 , srv347107890:84 , srv374515998:84 , srv388540800:84 , srv444446975:84 , srv454189076:84 , srv456128204:84 , srv478553441:84 , srv482168723:84 , srv485036057:84 , srv493551234:84 , srv497605619:84 , srv552095531:84 , srv581963227:84 , srv591966381:84 , srv62266559:84 , srv718691218:84 , srv726262841:84 , srv7371598:84 , srv753339350:84 , srv759404409:84 , srv797324978:84 , srv804853108:84 , srv893322009:84 , srv909215918:84 , srv961177573:84 , srv979738034:84 , srv989211243:84 , srv1798458872:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-03-24 13:53:28,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:28,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1092967907=5, srv14698537=26, srv1749496064=46, srv1726907993=45, srv347107890=73, srv1560012132=33, srv456128204=78, srv1688881717=44, srv1643119471=38, srv2075732596=64, srv374515998=74, srv7371598=90, srv1625887396=36, srv497605619=83, srv2004499467=57, srv1217889997=12, srv1780650489=47, srv1488839012=28, srv1635668467=37, srv1341984421=20, srv1452227652=25, srv1915571616=52, srv184793716=49, srv113056904=7, srv1388622987=22, srv893322009=95, srv1912798700=51, srv1646989268=40, srv1070227681=3, srv101698354=1, srv493551234=82, srv2062164577=63, srv759404409=92, srv1536043520=30, srv1104483018=6, srv2027701415=60, srv753339350=91, srv1659273052=42, srv62266559=87, srv1620221588=35, srv291180855=71, srv718691218=88, srv2118644438=66, srv444446975=76, srv388540800=75, srv1281478803=15, srv1180745137=9, srv1324011916=18, srv797324978=93, srv1798458872=48, srv591966381=86, srv1190790200=10, srv1553181010=31, srv581963227=85, srv1399881111=24, srv804853108=94, srv1395599617=23, srv281106914=70, srv1966279711=54, srv1061407474=2, srv1249407175=14, srv1554050261=32, srv485036057=81, srv979738034=98, srv235345887=67, srv909215918=96, srv1947001652=53, srv1160728679=8, srv1910885756=50, srv133637566=19, srv1644465393=39, srv1218301266=13, srv2114629450=65, srv1285519461=16, srv1983573527=55, srv198651878=56, srv1584420490=34, srv204462244=62, srv1300890373=17, srv726262841=89, srv1386103153=21, srv454189076=77, srv1085820900=4, srv1649912388=41, srv25056210=68, srv270456563=69, srv989211243=99, srv1675683664=43, srv1506260242=29, srv552095531=84, srv1201751666=11, srv1470513626=27, srv478553441=79, srv30665337=72, srv2009815200=58, srv482168723=80, srv961177573=97, srv2039703914=61, srv1008821124=0, srv2025105443=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:28,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:28,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-24 13:53:28,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-24 13:53:28,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-24 13:53:28,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-24 13:53:28,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-24 13:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-24 13:53:28,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-24 13:53:28,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-24 13:53:28,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-24 13:53:28,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-24 13:53:28,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-24 13:53:28,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:28,764 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-24 13:53:28,764 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-24 13:53:29,091 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 568 ms to try 5000 different iterations. Found a solution that moves 4684 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17838437626966722. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040040508232770916, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05367315978036025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5204444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040040508232770916, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17375, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:29,110 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1536043520:80 , srv1659273052:80 , srv1324011916:81 , srv1553181010:81 , srv1675683664:81 , srv581963227:81 , srv1643119471:82 , srv1688881717:82 , srv1947001652:82 , srv388540800:82 , srv797324978:82 , srv1201751666:83 , srv1388622987:83 , srv1470513626:83 , srv1560012132:83 , srv1983573527:83 , srv2004499467:83 , srv2062164577:83 , srv291180855:83 , srv62266559:83 , srv726262841:83 , srv7371598:83 , srv1061407474:84 , srv1249407175:84 , srv1452227652:84 , srv1749496064:84 , srv2114629450:84 , srv444446975:84 , srv753339350:84 , srv804853108:84 , srv989211243:84 , srv101698354:85 , srv1180745137:85 , srv1620221588:85 , srv1644465393:85 , srv1649912388:85 , srv1912798700:85 , srv198651878:85 , srv2027701415:85 , srv204462244:85 , srv25056210:85 , srv281106914:85 , srv347107890:85 , srv454189076:85 , srv591966381:85 , srv1218301266:86 , srv1395599617:86 , srv1506260242:86 , srv1625887396:86 , srv2025105443:86 , srv2039703914:86 , srv2118644438:86 , srv478553441:86 , srv497605619:86 , srv909215918:86 , srv1008821124:87 , srv1104483018:87 , srv1300890373:87 , srv1341984421:87 , srv1399881111:87 , srv1780650489:87 , srv456128204:87 , srv893322009:87 , srv1085820900:88 , srv1386103153:88 , srv14698537:88 , srv1554050261:88 , srv1635668467:88 , srv1915571616:88 , srv30665337:88 , srv374515998:88 , srv485036057:88 , srv493551234:88 , srv1217889997:89 , srv133637566:89 , srv1584420490:89 , srv1726907993:89 , srv1910885756:89 , srv235345887:89 , srv718691218:89 , srv759404409:89 , srv961177573:89 , srv1070227681:90 , srv1092967907:90 , srv1281478803:90 , srv1646989268:90 , srv184793716:90 , srv1966279711:90 , srv2009815200:90 , srv270456563:90 , srv979738034:90 , srv1160728679:91 , srv113056904:93 , srv1488839012:93 , srv1285519461:94 , srv482168723:94 , srv1190790200:95 , srv552095531:95 , srv2075732596:100 , srv1798458872:445 } 2024-03-24 13:53:29,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:29,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1092967907=5, srv14698537=26, srv1749496064=46, srv1726907993=45, srv347107890=73, srv1560012132=33, srv456128204=78, srv1688881717=44, srv1643119471=38, srv2075732596=64, srv374515998=74, srv7371598=90, srv1625887396=36, srv497605619=83, srv2004499467=57, srv1217889997=12, srv1780650489=47, srv1488839012=28, srv1635668467=37, srv1341984421=20, srv1452227652=25, srv1915571616=52, srv184793716=49, srv113056904=7, srv1388622987=22, srv893322009=95, srv1912798700=51, srv1646989268=40, srv1070227681=3, srv101698354=1, srv493551234=82, srv2062164577=63, srv759404409=92, srv1536043520=30, srv1104483018=6, srv2027701415=60, srv753339350=91, srv1659273052=42, srv62266559=87, srv1620221588=35, srv291180855=71, srv718691218=88, srv2118644438=66, srv444446975=76, srv388540800=75, srv1281478803=15, srv1180745137=9, srv1324011916=18, srv797324978=93, srv1798458872=48, srv591966381=86, srv1190790200=10, srv1553181010=31, srv581963227=85, srv1399881111=24, srv804853108=94, srv1395599617=23, srv281106914=70, srv1966279711=54, srv1061407474=2, srv1249407175=14, srv1554050261=32, srv485036057=81, srv979738034=98, srv235345887=67, srv909215918=96, srv1947001652=53, srv1160728679=8, srv1910885756=50, srv133637566=19, srv1644465393=39, srv1218301266=13, srv2114629450=65, srv1285519461=16, srv1983573527=55, srv198651878=56, srv1584420490=34, srv204462244=62, srv1300890373=17, srv726262841=89, srv1386103153=21, srv454189076=77, srv1085820900=4, srv1649912388=41, srv25056210=68, srv270456563=69, srv989211243=99, srv1675683664=43, srv1506260242=29, srv552095531=84, srv1201751666=11, srv1470513626=27, srv478553441=79, srv30665337=72, srv2009815200=58, srv482168723=80, srv961177573=97, srv2039703914=61, srv1008821124=0, srv2025105443=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:29,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:29,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:29,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:29,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:29,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:29,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-24 13:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-24 13:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-24 13:53:29,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-24 13:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-24 13:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-24 13:53:29,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-24 13:53:29,281 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040040508232770916, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05367315978036025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040040508232770916, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17375, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:29,282 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-24 13:53:29,282 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17835157820066, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040040508232770916, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05367315978036025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040040508232770916, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17375, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-24 13:53:29,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 350 ms to try 5000 different iterations. Found a solution that moves 2659 regions; Going from a computed imbalance of 0.17835157820066 to a new imbalance of 0.01444188122284186. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019818983796301076); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03131247962827889, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29544444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019818983796301076); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:29,510 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1285519461:76 , srv2075732596:76 , srv552095531:77 , srv113056904:78 , srv1488839012:78 , srv1190790200:79 , srv1092967907:80 , srv482168723:80 , srv1160728679:81 , srv1281478803:82 , srv1726907993:82 , srv1910885756:82 , srv1386103153:83 , srv1399881111:83 , srv14698537:83 , srv184793716:83 , srv1915571616:83 , srv1966279711:83 , srv270456563:83 , srv1070227681:84 , srv1085820900:84 , srv1217889997:84 , srv1300890373:84 , srv1506260242:84 , srv1554050261:84 , srv1584420490:84 , srv2009815200:84 , srv235345887:84 , srv456128204:84 , srv478553441:84 , srv485036057:84 , srv493551234:84 , srv718691218:84 , srv1008821124:85 , srv101698354:85 , srv1341984421:85 , srv1635668467:85 , srv1646989268:85 , srv1780650489:85 , srv2025105443:85 , srv25056210:85 , srv374515998:85 , srv961177573:85 , srv979738034:85 , srv1104483018:86 , srv133637566:86 , srv1395599617:86 , srv1452227652:86 , srv1620221588:86 , srv1625887396:86 , srv1649912388:86 , srv198651878:86 , srv204462244:86 , srv2114629450:86 , srv30665337:86 , srv1201751666:87 , srv1560012132:87 , srv1644465393:87 , srv1912798700:87 , srv2039703914:87 , srv281106914:87 , srv497605619:87 , srv804853108:87 , srv893322009:87 , srv909215918:87 , srv1180745137:88 , srv1218301266:88 , srv1249407175:88 , srv1388622987:88 , srv1749496064:88 , srv1983573527:88 , srv2118644438:88 , srv291180855:88 , srv347107890:88 , srv454189076:88 , srv726262841:88 , srv753339350:88 , srv759404409:88 , srv989211243:88 , srv1675683664:89 , srv2004499467:89 , srv444446975:89 , srv591966381:89 , srv62266559:89 , srv1061407474:90 , srv1470513626:90 , srv1643119471:90 , srv1688881717:90 , srv1947001652:90 , srv2027701415:90 , srv2062164577:90 , srv388540800:90 , srv7371598:90 , srv797324978:90 , srv1536043520:91 , srv581963227:91 , srv1324011916:92 , srv1659273052:92 , srv1553181010:93 , srv1798458872:505 } 2024-03-24 13:53:29,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:29,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1092967907=5, srv14698537=26, srv1749496064=46, srv1726907993=45, srv347107890=73, srv1560012132=33, srv456128204=78, srv1688881717=44, srv1643119471=38, srv2075732596=64, srv374515998=74, srv7371598=90, srv1625887396=36, srv497605619=83, srv2004499467=57, srv1217889997=12, srv1780650489=47, srv1488839012=28, srv1635668467=37, srv1341984421=20, srv1452227652=25, srv1915571616=52, srv184793716=49, srv113056904=7, srv1388622987=22, srv893322009=95, srv1912798700=51, srv1646989268=40, srv1070227681=3, srv101698354=1, srv493551234=82, srv2062164577=63, srv759404409=92, srv1536043520=30, srv1104483018=6, srv2027701415=60, srv753339350=91, srv1659273052=42, srv62266559=87, srv1620221588=35, srv291180855=71, srv718691218=88, srv2118644438=66, srv444446975=76, srv388540800=75, srv1281478803=15, srv1180745137=9, srv1324011916=18, srv797324978=93, srv1798458872=48, srv591966381=86, srv1190790200=10, srv1553181010=31, srv581963227=85, srv1399881111=24, srv804853108=94, srv1395599617=23, srv281106914=70, srv1966279711=54, srv1061407474=2, srv1249407175=14, srv1554050261=32, srv485036057=81, srv979738034=98, srv235345887=67, srv909215918=96, srv1947001652=53, srv1160728679=8, srv1910885756=50, srv133637566=19, srv1644465393=39, srv1218301266=13, srv2114629450=65, srv1285519461=16, srv1983573527=55, srv198651878=56, srv1584420490=34, srv204462244=62, srv1300890373=17, srv726262841=89, srv1386103153=21, srv454189076=77, srv1085820900=4, srv1649912388=41, srv25056210=68, srv270456563=69, srv989211243=99, srv1675683664=43, srv1506260242=29, srv552095531=84, srv1201751666=11, srv1470513626=27, srv478553441=79, srv30665337=72, srv2009815200=58, srv482168723=80, srv961177573=97, srv2039703914=61, srv1008821124=0, srv2025105443=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-24 13:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-24 13:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-24 13:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-24 13:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-24 13:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-24 13:53:29,686 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019818983796301076); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03131247962827889, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019818983796301076); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:29,686 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-24 13:53:29,686 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014423262506896063, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019818983796301076); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03131247962827889, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019818983796301076); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-24 13:53:29,908 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 351 ms to try 5000 different iterations. Found a solution that moves 994 regions; Going from a computed imbalance of 0.014423262506896063 to a new imbalance of 5.245398733062647E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004424760000269228); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.11044444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004424760000269228); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:29,909 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv759404409:81 , srv133637566:82 , srv2075732596:82 , srv482168723:82 , srv1061407474:83 , srv1190790200:83 , srv1553181010:83 , srv1646989268:83 , srv2009815200:83 , srv2027701415:83 , srv2118644438:83 , srv591966381:83 , srv961177573:83 , srv979738034:83 , srv1070227681:84 , srv1180745137:84 , srv1324011916:84 , srv1584420490:84 , srv1966279711:84 , srv2039703914:84 , srv30665337:84 , srv893322009:84 , srv909215918:84 , srv1008821124:85 , srv1104483018:85 , srv1217889997:85 , srv1218301266:85 , srv1395599617:85 , srv1470513626:85 , srv1635668467:85 , srv1644465393:85 , srv1659273052:85 , srv1688881717:85 , srv1749496064:85 , srv184793716:85 , srv1947001652:85 , srv2062164577:85 , srv235345887:85 , srv270456563:85 , srv347107890:85 , srv374515998:85 , srv444446975:85 , srv454189076:85 , srv581963227:85 , srv718691218:85 , srv7371598:85 , srv797324978:85 , srv1085820900:86 , srv1160728679:86 , srv1249407175:86 , srv1281478803:86 , srv1300890373:86 , srv1341984421:86 , srv1388622987:86 , srv14698537:86 , srv1488839012:86 , srv1536043520:86 , srv1554050261:86 , srv1625887396:86 , srv1643119471:86 , srv1780650489:86 , srv1910885756:86 , srv1912798700:86 , srv1915571616:86 , srv1983573527:86 , srv198651878:86 , srv2004499467:86 , srv2025105443:86 , srv281106914:86 , srv388540800:86 , srv485036057:86 , srv497605619:86 , srv552095531:86 , srv62266559:86 , srv726262841:86 , srv753339350:86 , srv989211243:86 , srv101698354:87 , srv113056904:87 , srv1285519461:87 , srv1386103153:87 , srv1452227652:87 , srv1506260242:87 , srv1560012132:87 , srv1620221588:87 , srv1649912388:87 , srv1726907993:87 , srv204462244:87 , srv25056210:87 , srv456128204:87 , srv478553441:87 , srv493551234:87 , srv804853108:87 , srv1092967907:88 , srv1201751666:88 , srv1399881111:88 , srv1675683664:88 , srv2114629450:88 , srv291180855:88 , srv1798458872:547 } 2024-03-24 13:53:29,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:29,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1092967907=5, srv14698537=26, srv1749496064=46, srv1726907993=45, srv347107890=73, srv1560012132=33, srv456128204=78, srv1688881717=44, srv1643119471=38, srv2075732596=64, srv374515998=74, srv7371598=90, srv1625887396=36, srv497605619=83, srv2004499467=57, srv1217889997=12, srv1780650489=47, srv1488839012=28, srv1635668467=37, srv1341984421=20, srv1452227652=25, srv1915571616=52, srv184793716=49, srv113056904=7, srv1388622987=22, srv893322009=95, srv1912798700=51, srv1646989268=40, srv1070227681=3, srv101698354=1, srv493551234=82, srv2062164577=63, srv759404409=92, srv1536043520=30, srv1104483018=6, srv2027701415=60, srv753339350=91, srv1659273052=42, srv62266559=87, srv1620221588=35, srv291180855=71, srv718691218=88, srv2118644438=66, srv444446975=76, srv388540800=75, srv1281478803=15, srv1180745137=9, srv1324011916=18, srv797324978=93, srv1798458872=48, srv591966381=86, srv1190790200=10, srv1553181010=31, srv581963227=85, srv1399881111=24, srv804853108=94, srv1395599617=23, srv281106914=70, srv1966279711=54, srv1061407474=2, srv1249407175=14, srv1554050261=32, srv485036057=81, srv979738034=98, srv235345887=67, srv909215918=96, srv1947001652=53, srv1160728679=8, srv1910885756=50, srv133637566=19, srv1644465393=39, srv1218301266=13, srv2114629450=65, srv1285519461=16, srv1983573527=55, srv198651878=56, srv1584420490=34, srv204462244=62, srv1300890373=17, srv726262841=89, srv1386103153=21, srv454189076=77, srv1085820900=4, srv1649912388=41, srv25056210=68, srv270456563=69, srv989211243=99, srv1675683664=43, srv1506260242=29, srv552095531=84, srv1201751666=11, srv1470513626=27, srv478553441=79, srv30665337=72, srv2009815200=58, srv482168723=80, srv961177573=97, srv2039703914=61, srv1008821124=0, srv2025105443=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:29,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:29,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:29,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:29,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-24 13:53:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-24 13:53:29,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-24 13:53:29,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-24 13:53:29,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-24 13:53:29,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-24 13:53:30,035 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004424760000269228); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004424760000269228); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:30,035 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-24 13:53:30,036 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.175797365442788E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004424760000269228); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004424760000269228); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-24 13:53:30,365 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 425 ms to try 5000 different iterations. Found a solution that moves 232 regions; Going from a computed imbalance of 5.175797365442788E-4 to a new imbalance of 2.7172241865993972E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.025777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:30,366 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv497605619:82 , srv1218301266:83 , srv2004499467:83 , srv493551234:83 , srv718691218:83 , srv1070227681:84 , srv1085820900:84 , srv1092967907:84 , srv1104483018:84 , srv113056904:84 , srv1160728679:84 , srv1201751666:84 , srv1217889997:84 , srv1249407175:84 , srv1281478803:84 , srv1341984421:84 , srv1386103153:84 , srv1388622987:84 , srv1399881111:84 , srv1470513626:84 , srv1536043520:84 , srv1554050261:84 , srv1584420490:84 , srv1620221588:84 , srv1625887396:84 , srv1635668467:84 , srv1643119471:84 , srv1646989268:84 , srv1649912388:84 , srv1675683664:84 , srv1726907993:84 , srv1780650489:84 , srv184793716:84 , srv1912798700:84 , srv1947001652:84 , srv204462244:84 , srv2075732596:84 , srv2114629450:84 , srv235345887:84 , srv270456563:84 , srv281106914:84 , srv291180855:84 , srv30665337:84 , srv347107890:84 , srv374515998:84 , srv388540800:84 , srv444446975:84 , srv454189076:84 , srv456128204:84 , srv485036057:84 , srv552095531:84 , srv62266559:84 , srv7371598:84 , srv759404409:84 , srv804853108:84 , srv893322009:84 , srv979738034:84 , srv989211243:84 , srv1008821124:85 , srv101698354:85 , srv1061407474:85 , srv1180745137:85 , srv1190790200:85 , srv1285519461:85 , srv1300890373:85 , srv1324011916:85 , srv1395599617:85 , srv1452227652:85 , srv14698537:85 , srv1506260242:85 , srv1553181010:85 , srv1560012132:85 , srv1644465393:85 , srv1659273052:85 , srv1688881717:85 , srv1749496064:85 , srv1910885756:85 , srv1915571616:85 , srv1983573527:85 , srv198651878:85 , srv2009815200:85 , srv2025105443:85 , srv2027701415:85 , srv2039703914:85 , srv2062164577:85 , srv2118644438:85 , srv25056210:85 , srv478553441:85 , srv581963227:85 , srv591966381:85 , srv726262841:85 , srv753339350:85 , srv797324978:85 , srv909215918:85 , srv961177573:85 , srv133637566:86 , srv1488839012:86 , srv1966279711:86 , srv482168723:86 , srv1798458872:645 } 2024-03-24 13:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1092967907=5, srv14698537=26, srv1749496064=46, srv1726907993=45, srv347107890=73, srv1560012132=33, srv456128204=78, srv1688881717=44, srv1643119471=38, srv2075732596=64, srv374515998=74, srv7371598=90, srv1625887396=36, srv497605619=83, srv2004499467=57, srv1217889997=12, srv1780650489=47, srv1488839012=28, srv1635668467=37, srv1341984421=20, srv1452227652=25, srv1915571616=52, srv184793716=49, srv113056904=7, srv1388622987=22, srv893322009=95, srv1912798700=51, srv1646989268=40, srv1070227681=3, srv101698354=1, srv493551234=82, srv2062164577=63, srv759404409=92, srv1536043520=30, srv1104483018=6, srv2027701415=60, srv753339350=91, srv1659273052=42, srv62266559=87, srv1620221588=35, srv291180855=71, srv718691218=88, srv2118644438=66, srv444446975=76, srv388540800=75, srv1281478803=15, srv1180745137=9, srv1324011916=18, srv797324978=93, srv1798458872=48, srv591966381=86, srv1190790200=10, srv1553181010=31, srv581963227=85, srv1399881111=24, srv804853108=94, srv1395599617=23, srv281106914=70, srv1966279711=54, srv1061407474=2, srv1249407175=14, srv1554050261=32, srv485036057=81, srv979738034=98, srv235345887=67, srv909215918=96, srv1947001652=53, srv1160728679=8, srv1910885756=50, srv133637566=19, srv1644465393=39, srv1218301266=13, srv2114629450=65, srv1285519461=16, srv1983573527=55, srv198651878=56, srv1584420490=34, srv204462244=62, srv1300890373=17, srv726262841=89, srv1386103153=21, srv454189076=77, srv1085820900=4, srv1649912388=41, srv25056210=68, srv270456563=69, srv989211243=99, srv1675683664=43, srv1506260242=29, srv552095531=84, srv1201751666=11, srv1470513626=27, srv478553441=79, srv30665337=72, srv2009815200=58, srv482168723=80, srv961177573=97, srv2039703914=61, srv1008821124=0, srv2025105443=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:30,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:30,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-24 13:53:30,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-24 13:53:30,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-24 13:53:30,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-24 13:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-24 13:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-24 13:53:30,490 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:30,490 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-24 13:53:30,490 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.5547743144886592E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-24 13:53:30,713 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 318 ms to try 5000 different iterations. Found a solution that moves 22 regions; Going from a computed imbalance of 2.5547743144886592E-5 to a new imbalance of 6.898116346350751E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0024444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:30,714 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv133637566:83 , srv1488839012:83 , srv1966279711:83 , srv2027701415:83 , srv2062164577:83 , srv291180855:83 , srv753339350:83 , srv1008821124:84 , srv101698354:84 , srv1061407474:84 , srv1070227681:84 , srv1085820900:84 , srv1092967907:84 , srv1104483018:84 , srv113056904:84 , srv1160728679:84 , srv1180745137:84 , srv1190790200:84 , srv1201751666:84 , srv1217889997:84 , srv1218301266:84 , srv1249407175:84 , srv1281478803:84 , srv1285519461:84 , srv1300890373:84 , srv1324011916:84 , srv1341984421:84 , srv1386103153:84 , srv1395599617:84 , srv1399881111:84 , srv1452227652:84 , srv14698537:84 , srv1470513626:84 , srv1506260242:84 , srv1553181010:84 , srv1554050261:84 , srv1560012132:84 , srv1620221588:84 , srv1625887396:84 , srv1635668467:84 , srv1643119471:84 , srv1644465393:84 , srv1646989268:84 , srv1649912388:84 , srv1659273052:84 , srv1675683664:84 , srv1688881717:84 , srv1726907993:84 , srv1749496064:84 , srv1780650489:84 , srv184793716:84 , srv1910885756:84 , srv1912798700:84 , srv1915571616:84 , srv1983573527:84 , srv198651878:84 , srv2009815200:84 , srv2025105443:84 , srv2039703914:84 , srv204462244:84 , srv2075732596:84 , srv2114629450:84 , srv2118644438:84 , srv235345887:84 , srv25056210:84 , srv270456563:84 , srv281106914:84 , srv30665337:84 , srv347107890:84 , srv374515998:84 , srv388540800:84 , srv444446975:84 , srv454189076:84 , srv456128204:84 , srv478553441:84 , srv482168723:84 , srv485036057:84 , srv493551234:84 , srv552095531:84 , srv581963227:84 , srv591966381:84 , srv62266559:84 , srv726262841:84 , srv7371598:84 , srv759404409:84 , srv797324978:84 , srv804853108:84 , srv893322009:84 , srv909215918:84 , srv961177573:84 , srv979738034:84 , srv989211243:84 , srv1388622987:85 , srv1536043520:85 , srv1584420490:85 , srv1947001652:85 , srv2004499467:85 , srv497605619:85 , srv718691218:85 , srv1798458872:684 } 2024-03-24 13:53:30,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:30,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1092967907=5, srv14698537=26, srv1749496064=46, srv1726907993=45, srv347107890=73, srv1560012132=33, srv456128204=78, srv1688881717=44, srv1643119471=38, srv2075732596=64, srv374515998=74, srv7371598=90, srv1625887396=36, srv497605619=83, srv2004499467=57, srv1217889997=12, srv1780650489=47, srv1488839012=28, srv1635668467=37, srv1341984421=20, srv1452227652=25, srv1915571616=52, srv184793716=49, srv113056904=7, srv1388622987=22, srv893322009=95, srv1912798700=51, srv1646989268=40, srv1070227681=3, srv101698354=1, srv493551234=82, srv2062164577=63, srv759404409=92, srv1536043520=30, srv1104483018=6, srv2027701415=60, srv753339350=91, srv1659273052=42, srv62266559=87, srv1620221588=35, srv291180855=71, srv718691218=88, srv2118644438=66, srv444446975=76, srv388540800=75, srv1281478803=15, srv1180745137=9, srv1324011916=18, srv797324978=93, srv1798458872=48, srv591966381=86, srv1190790200=10, srv1553181010=31, srv581963227=85, srv1399881111=24, srv804853108=94, srv1395599617=23, srv281106914=70, srv1966279711=54, srv1061407474=2, srv1249407175=14, srv1554050261=32, srv485036057=81, srv979738034=98, srv235345887=67, srv909215918=96, srv1947001652=53, srv1160728679=8, srv1910885756=50, srv133637566=19, srv1644465393=39, srv1218301266=13, srv2114629450=65, srv1285519461=16, srv1983573527=55, srv198651878=56, srv1584420490=34, srv204462244=62, srv1300890373=17, srv726262841=89, srv1386103153=21, srv454189076=77, srv1085820900=4, srv1649912388=41, srv25056210=68, srv270456563=69, srv989211243=99, srv1675683664=43, srv1506260242=29, srv552095531=84, srv1201751666=11, srv1470513626=27, srv478553441=79, srv30665337=72, srv2009815200=58, srv482168723=80, srv961177573=97, srv2039703914=61, srv1008821124=0, srv2025105443=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-24 13:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-24 13:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-24 13:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-24 13:53:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-24 13:53:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-24 13:53:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-24 13:53:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-24 13:53:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-24 13:53:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-24 13:53:30,848 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:30,849 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv133637566:83 , srv1488839012:83 , srv1966279711:83 , srv2027701415:83 , srv2062164577:83 , srv291180855:83 , srv753339350:83 , srv1008821124:84 , srv101698354:84 , srv1061407474:84 , srv1070227681:84 , srv1085820900:84 , srv1092967907:84 , srv1104483018:84 , srv113056904:84 , srv1160728679:84 , srv1180745137:84 , srv1190790200:84 , srv1201751666:84 , srv1217889997:84 , srv1218301266:84 , srv1249407175:84 , srv1281478803:84 , srv1285519461:84 , srv1300890373:84 , srv1324011916:84 , srv1341984421:84 , srv1386103153:84 , srv1395599617:84 , srv1399881111:84 , srv1452227652:84 , srv14698537:84 , srv1470513626:84 , srv1506260242:84 , srv1553181010:84 , srv1554050261:84 , srv1560012132:84 , srv1620221588:84 , srv1625887396:84 , srv1635668467:84 , srv1643119471:84 , srv1644465393:84 , srv1646989268:84 , srv1649912388:84 , srv1659273052:84 , srv1675683664:84 , srv1688881717:84 , srv1726907993:84 , srv1749496064:84 , srv1780650489:84 , srv184793716:84 , srv1910885756:84 , srv1912798700:84 , srv1915571616:84 , srv1983573527:84 , srv198651878:84 , srv2009815200:84 , srv2025105443:84 , srv2039703914:84 , srv204462244:84 , srv2075732596:84 , srv2114629450:84 , srv2118644438:84 , srv235345887:84 , srv25056210:84 , srv270456563:84 , srv281106914:84 , srv30665337:84 , srv347107890:84 , srv374515998:84 , srv388540800:84 , srv444446975:84 , srv454189076:84 , srv456128204:84 , srv478553441:84 , srv482168723:84 , srv485036057:84 , srv493551234:84 , srv552095531:84 , srv581963227:84 , srv591966381:84 , srv62266559:84 , srv726262841:84 , srv7371598:84 , srv759404409:84 , srv797324978:84 , srv804853108:84 , srv893322009:84 , srv909215918:84 , srv961177573:84 , srv979738034:84 , srv989211243:84 , srv1388622987:85 , srv1536043520:85 , srv1584420490:85 , srv1947001652:85 , srv2004499467:85 , srv497605619:85 , srv718691218:85 , srv1798458872:684 } 2024-03-24 13:53:30,876 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=170 (was 170), AvailableMemoryMB=13456 (was 13967) 2024-03-24 13:53:30,885 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=170, AvailableMemoryMB=13456 2024-03-24 13:53:30,886 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-03-24 13:53:30,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-03-24 13:53:30,898 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1054269086:84 , srv1092894445:84 , srv1127796982:84 , srv1270224330:84 , srv133712467:84 , srv1364353967:84 , srv1510052529:84 , srv1577988449:84 , srv1669284636:84 , srv1691242048:84 , srv1701437446:84 , srv1712502514:84 , srv1903741591:84 , srv191013957:84 , srv2003652212:84 , srv2082172378:84 , srv2096322951:84 , srv287787548:84 , srv288826194:84 , srv329928872:84 , srv427359880:84 , srv537234224:84 , srv601905502:84 , srv639954685:84 , srv797671875:84 , srv819956721:84 , srv856612038:84 , srv888286857:84 , srv923096432:84 , srv953384390:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-03-24 13:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1364353967=5, srv1510052529=6, srv1054269086=0, srv287787548=17, srv191013957=13, srv1270224330=3, srv288826194=18, srv797671875=24, srv2003652212=14, srv953384390=29, srv1092894445=1, srv1712502514=11, srv888286857=27, srv601905502=22, srv133712467=4, srv1691242048=9, srv1669284636=8, srv1127796982=2, srv329928872=19, srv1903741591=12, srv1701437446=10, srv923096432=28, srv2082172378=15, srv2096322951=16, srv537234224=21, srv639954685=23, srv856612038=26, srv427359880=20, srv819956721=25, srv1577988449=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:53:30,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:53:30,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:53:30,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:53:30,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-03-24 13:53:30,942 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.104714963874858, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:53:30,943 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912522996696191, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.104714963874858, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-03-24 13:55:30,920 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6207947 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912522996696191 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:55:30,926 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1054269086:90 , srv1092894445:90 , srv1127796982:90 , srv1270224330:90 , srv133712467:90 , srv1364353967:90 , srv1510052529:90 , srv1577988449:90 , srv1669284636:90 , srv1691242048:90 , srv1701437446:90 , srv1712502514:90 , srv1903741591:90 , srv191013957:90 , srv2003652212:90 , srv2082172378:90 , srv2096322951:90 , srv287787548:90 , srv288826194:90 , srv329928872:90 , srv427359880:90 , srv537234224:90 , srv601905502:90 , srv639954685:90 , srv797671875:90 , srv819956721:90 , srv856612038:90 , srv888286857:90 , srv923096432:90 , srv953384390:90 } 2024-03-24 13:55:30,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-24 13:55:30,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1364353967=5, srv1510052529=6, srv1054269086=0, srv287787548=17, srv191013957=13, srv1270224330=3, srv288826194=18, srv797671875=24, srv2003652212=14, srv953384390=29, srv1092894445=1, srv1712502514=11, srv888286857=27, srv601905502=22, srv133712467=4, srv1691242048=9, srv1669284636=8, srv1127796982=2, srv329928872=19, srv1903741591=12, srv1701437446=10, srv923096432=28, srv2082172378=15, srv2096322951=16, srv537234224=21, srv639954685=23, srv856612038=26, srv427359880=20, srv819956721=25, srv1577988449=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-24 13:55:30,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-24 13:55:30,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-24 13:55:30,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-03-24 13:55:30,943 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-24 13:55:30,944 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1054269086:90 , srv1092894445:90 , srv1127796982:90 , srv1270224330:90 , srv133712467:90 , srv1364353967:90 , srv1510052529:90 , srv1577988449:90 , srv1669284636:90 , srv1691242048:90 , srv1701437446:90 , srv1712502514:90 , srv1903741591:90 , srv191013957:90 , srv2003652212:90 , srv2082172378:90 , srv2096322951:90 , srv287787548:90 , srv288826194:90 , srv329928872:90 , srv427359880:90 , srv537234224:90 , srv601905502:90 , srv639954685:90 , srv797671875:90 , srv819956721:90 , srv856612038:90 , srv888286857:90 , srv923096432:90 , srv953384390:90 } 2024-03-24 13:55:30,958 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=164 (was 242), ProcessCount=168 (was 170), AvailableMemoryMB=12486 (was 13456)