2023-08-31 08:54:20,003 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-31 08:54:20,339 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-31 08:54:20,452 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-31 08:54:20,515 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-31 08:54:20,524 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-31 08:54:20,538 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=167, AvailableMemoryMB=10775 2023-08-31 08:54:20,540 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-31 08:54:20,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-31 08:54:20,708 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026068441:84 , srv1034923926:84 , srv1046553179:84 , srv1068582366:84 , srv1078985917:84 , srv10905006:84 , srv109210577:84 , srv1119758026:84 , srv1136645188:84 , srv1136898638:84 , srv1193559265:84 , srv1214560921:84 , srv1271655892:84 , srv1292835889:84 , srv1314861810:84 , srv1331104194:84 , srv1331901251:84 , srv1343649209:84 , srv1344969402:84 , srv1388779071:84 , srv1398583404:84 , srv1416132389:84 , srv1453401233:84 , srv1554130133:84 , srv1561000048:84 , srv1573939300:84 , srv1591149025:84 , srv161244321:84 , srv1613291777:84 , srv1624236161:84 , srv1632430351:84 , srv165898153:84 , srv1668615708:84 , srv1692009514:84 , srv1730446140:84 , srv1732054744:84 , srv1817623213:84 , srv1827068063:84 , srv1838108622:84 , srv1878294466:84 , srv1891815345:84 , srv1916240833:84 , srv1929547911:84 , srv1942672919:84 , srv194876440:84 , srv1976837660:84 , srv1976867333:84 , srv1978201535:84 , srv2007327392:84 , srv2050433505:84 , srv2064067865:84 , srv2084168914:84 , srv208509088:84 , srv2120632631:84 , srv2121160851:84 , srv2122485981:84 , srv224658972:84 , srv252616210:84 , srv289635978:84 , srv309207915:84 , srv344601824:84 , srv347255023:84 , srv348493608:84 , srv393951548:84 , srv419126424:84 , srv470128112:84 , srv501724490:84 , srv504931757:84 , srv533074508:84 , srv54178412:84 , srv565958855:84 , srv580749837:84 , srv583522121:84 , srv583701446:84 , srv618018085:84 , srv62708963:84 , srv636148501:84 , srv665256066:84 , srv669538430:84 , srv677918921:84 , srv700800755:84 , srv746202407:84 , srv750263286:84 , srv755976783:84 , srv767080301:84 , srv807095526:84 , srv824690011:84 , srv835131595:84 , srv835461685:84 , srv848576952:84 , srv848915681:84 , srv85647128:84 , srv868891094:84 , srv876085335:84 , srv919529277:84 , srv928458748:84 , srv938063459:84 , srv982349716:84 , srv99878656:84 , srv1881342767:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-31 08:54:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:20,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv194876440=45, srv750263286=83, srv1632430351=30, srv10905006=5, srv2007327392=49, srv393951548=64, srv2120632631=54, srv1136898638=9, srv807095526=86, srv700800755=81, srv533074508=69, srv1193559265=10, srv1730446140=34, srv1613291777=28, srv1878294466=39, srv2121160851=55, srv208509088=53, srv665256066=78, srv876085335=94, srv1344969402=18, srv1978201535=48, srv868891094=93, srv1976837660=46, srv565958855=71, srv919529277=95, srv1214560921=11, srv1827068063=37, srv583701446=74, srv1624236161=29, srv2050433505=50, srv636148501=77, srv1292835889=13, srv1314861810=14, srv767080301=85, srv504931757=68, srv1929547911=43, srv1692009514=33, srv1119758026=7, srv419126424=65, srv348493608=63, srv618018085=75, srv1416132389=21, srv252616210=58, srv289635978=59, srv1331104194=15, srv1817623213=36, srv1976867333=47, srv2122485981=56, srv928458748=96, srv1891815345=41, srv344601824=61, srv938063459=97, srv1554130133=23, srv224658972=57, srv1453401233=22, srv1271655892=12, srv309207915=60, srv54178412=70, srv62708963=76, srv1881342767=40, srv669538430=79, srv347255023=62, srv1078985917=4, srv1561000048=24, srv85647128=92, srv1838108622=38, srv161244321=27, srv583522121=73, srv1732054744=35, srv746202407=82, srv99878656=99, srv1668615708=32, srv835461685=89, srv1398583404=20, srv982349716=98, srv1331901251=16, srv848915681=91, srv1034923926=1, srv755976783=84, srv1046553179=2, srv2084168914=52, srv677918921=80, srv1068582366=3, srv1136645188=8, srv848576952=90, srv1388779071=19, srv824690011=87, srv1591149025=26, srv2064067865=51, srv1026068441=0, srv470128112=66, srv580749837=72, srv835131595=88, srv109210577=6, srv1942672919=44, srv1573939300=25, srv1343649209=17, srv1916240833=42, srv501724490=67, srv165898153=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:20,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:20,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:20,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:20,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:20,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:20,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:20,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:20,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:20,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:20,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:20,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:20,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:20,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:20,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:20,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-31 08:54:20,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-31 08:54:20,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-31 08:54:20,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-31 08:54:20,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-31 08:54:20,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-31 08:54:20,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-31 08:54:20,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-31 08:54:20,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-31 08:54:20,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-31 08:54:20,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-31 08:54:20,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-31 08:54:20,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-31 08:54:20,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-31 08:54:20,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-31 08:54:20,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-31 08:54:20,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-31 08:54:20,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-31 08:54:20,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-31 08:54:20,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-31 08:54:20,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-31 08:54:20,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-31 08:54:20,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-31 08:54:20,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-31 08:54:20,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-31 08:54:20,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-31 08:54:20,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-31 08:54:20,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-31 08:54:20,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-31 08:54:20,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-31 08:54:20,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-31 08:54:20,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-31 08:54:21,222 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:21,222 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-31 08:54:21,223 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-31 08:54:21,994 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1145 ms to try 5000 different iterations. Found a solution that moves 4707 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17899054203083203. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03873804318141119, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05635983478001844, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.523, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03873804318141119, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:22,014 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1136898638:81 , srv1453401233:81 , srv1891815345:81 , srv583701446:82 , srv746202407:82 , srv1034923926:83 , srv1119758026:83 , srv1632430351:83 , srv165898153:83 , srv2050433505:83 , srv208509088:83 , srv419126424:83 , srv501724490:83 , srv824690011:83 , srv1331901251:84 , srv1344969402:84 , srv1561000048:84 , srv1591149025:84 , srv1692009514:84 , srv1817623213:84 , srv1838108622:84 , srv252616210:84 , srv348493608:84 , srv393951548:84 , srv636148501:84 , srv109210577:85 , srv1331104194:85 , srv1388779071:85 , srv161244321:85 , srv1942672919:85 , srv1978201535:85 , srv767080301:85 , srv876085335:85 , srv919529277:85 , srv1068582366:86 , srv1292835889:86 , srv1398583404:86 , srv1416132389:86 , srv1624236161:86 , srv1916240833:86 , srv1929547911:86 , srv194876440:86 , srv1976837660:86 , srv2064067865:86 , srv224658972:86 , srv583522121:86 , srv618018085:86 , srv62708963:86 , srv665256066:86 , srv677918921:86 , srv755976783:86 , srv835461685:86 , srv868891094:86 , srv938063459:86 , srv10905006:87 , srv1668615708:87 , srv1730446140:87 , srv1827068063:87 , srv2121160851:87 , srv2122485981:87 , srv344601824:87 , srv470128112:87 , srv504931757:87 , srv533074508:87 , srv54178412:87 , srv807095526:87 , srv848576952:87 , srv848915681:87 , srv928458748:87 , srv1026068441:88 , srv1573939300:88 , srv289635978:88 , srv309207915:88 , srv347255023:88 , srv669538430:88 , srv700800755:88 , srv835131595:88 , srv99878656:88 , srv565958855:89 , srv750263286:89 , srv1046553179:90 , srv1078985917:90 , srv1193559265:90 , srv1314861810:90 , srv1343649209:90 , srv1613291777:90 , srv1878294466:90 , srv1976867333:90 , srv2120632631:90 , srv580749837:90 , srv982349716:90 , srv1136645188:91 , srv1214560921:91 , srv1732054744:91 , srv2007327392:91 , srv85647128:91 , srv2084168914:92 , srv1271655892:94 , srv1554130133:97 , srv1881342767:434 } 2023-08-31 08:54:22,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:22,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv194876440=45, srv750263286=83, srv1632430351=30, srv10905006=5, srv2007327392=49, srv393951548=64, srv2120632631=54, srv1136898638=9, srv807095526=86, srv700800755=81, srv533074508=69, srv1193559265=10, srv1730446140=34, srv1613291777=28, srv1878294466=39, srv2121160851=55, srv208509088=53, srv665256066=78, srv876085335=94, srv1344969402=18, srv1978201535=48, srv868891094=93, srv1976837660=46, srv565958855=71, srv919529277=95, srv1214560921=11, srv1827068063=37, srv583701446=74, srv1624236161=29, srv2050433505=50, srv636148501=77, srv1292835889=13, srv1314861810=14, srv767080301=85, srv504931757=68, srv1929547911=43, srv1692009514=33, srv1119758026=7, srv419126424=65, srv348493608=63, srv618018085=75, srv1416132389=21, srv252616210=58, srv289635978=59, srv1331104194=15, srv1817623213=36, srv1976867333=47, srv2122485981=56, srv928458748=96, srv1891815345=41, srv344601824=61, srv938063459=97, srv1554130133=23, srv224658972=57, srv1453401233=22, srv1271655892=12, srv309207915=60, srv54178412=70, srv62708963=76, srv1881342767=40, srv669538430=79, srv347255023=62, srv1078985917=4, srv1561000048=24, srv85647128=92, srv1838108622=38, srv161244321=27, srv583522121=73, srv1732054744=35, srv746202407=82, srv99878656=99, srv1668615708=32, srv835461685=89, srv1398583404=20, srv982349716=98, srv1331901251=16, srv848915681=91, srv1034923926=1, srv755976783=84, srv1046553179=2, srv2084168914=52, srv677918921=80, srv1068582366=3, srv1136645188=8, srv848576952=90, srv1388779071=19, srv824690011=87, srv1591149025=26, srv2064067865=51, srv1026068441=0, srv470128112=66, srv580749837=72, srv835131595=88, srv109210577=6, srv1942672919=44, srv1573939300=25, srv1343649209=17, srv1916240833=42, srv501724490=67, srv165898153=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:22,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:22,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:22,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:22,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-31 08:54:22,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-31 08:54:22,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-31 08:54:22,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-31 08:54:22,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-31 08:54:22,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-31 08:54:22,257 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03873804318141119, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05635983478001844, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03873804318141119, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:22,257 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-31 08:54:22,257 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17895758291238267, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03873804318141119, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05635983478001844, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03873804318141119, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-31 08:54:22,621 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 561 ms to try 5000 different iterations. Found a solution that moves 2621 regions; Going from a computed imbalance of 0.17895758291238267 to a new imbalance of 0.01267185186084729. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01892878865063798); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027438329994042148, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2912222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01892878865063798); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0095); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043166666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:22,631 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1271655892:76 , srv1554130133:77 , srv1732054744:80 , srv982349716:80 , srv1214560921:81 , srv1314861810:81 , srv2007327392:81 , srv1046553179:82 , srv565958855:82 , srv580749837:82 , srv700800755:82 , srv835131595:82 , srv85647128:82 , srv1078985917:83 , srv1136645188:83 , srv1193559265:83 , srv1343649209:83 , srv1613291777:83 , srv1878294466:83 , srv1976867333:83 , srv2120632631:83 , srv2121160851:83 , srv2122485981:83 , srv750263286:83 , srv807095526:83 , srv10905006:84 , srv1398583404:84 , srv2084168914:84 , srv289635978:84 , srv470128112:84 , srv533074508:84 , srv677918921:84 , srv868891094:84 , srv1668615708:85 , srv1827068063:85 , srv1916240833:85 , srv2064067865:85 , srv224658972:85 , srv309207915:85 , srv344601824:85 , srv504931757:85 , srv62708963:85 , srv755976783:85 , srv848576952:85 , srv938063459:85 , srv99878656:85 , srv1026068441:86 , srv1292835889:86 , srv1331104194:86 , srv1561000048:86 , srv161244321:86 , srv1624236161:86 , srv1730446140:86 , srv1942672919:86 , srv194876440:86 , srv1976837660:86 , srv347255023:86 , srv767080301:86 , srv835461685:86 , srv848915681:86 , srv1068582366:87 , srv109210577:87 , srv1416132389:87 , srv393951548:87 , srv618018085:87 , srv876085335:87 , srv919529277:87 , srv928458748:87 , srv1034923926:88 , srv1331901251:88 , srv1388779071:88 , srv1573939300:88 , srv165898153:88 , srv1817623213:88 , srv1929547911:88 , srv501724490:88 , srv54178412:88 , srv583522121:88 , srv636148501:88 , srv669538430:88 , srv1344969402:89 , srv1978201535:89 , srv348493608:89 , srv1119758026:90 , srv1453401233:90 , srv1632430351:90 , srv1692009514:90 , srv208509088:90 , srv252616210:90 , srv419126424:90 , srv665256066:90 , srv746202407:90 , srv824690011:90 , srv1838108622:91 , srv1891815345:91 , srv2050433505:91 , srv583701446:91 , srv1136898638:92 , srv1591149025:92 , srv1881342767:508 } 2023-08-31 08:54:22,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:22,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv194876440=45, srv750263286=83, srv1632430351=30, srv10905006=5, srv2007327392=49, srv393951548=64, srv2120632631=54, srv1136898638=9, srv807095526=86, srv700800755=81, srv533074508=69, srv1193559265=10, srv1730446140=34, srv1613291777=28, srv1878294466=39, srv2121160851=55, srv208509088=53, srv665256066=78, srv876085335=94, srv1344969402=18, srv1978201535=48, srv868891094=93, srv1976837660=46, srv565958855=71, srv919529277=95, srv1214560921=11, srv1827068063=37, srv583701446=74, srv1624236161=29, srv2050433505=50, srv636148501=77, srv1292835889=13, srv1314861810=14, srv767080301=85, srv504931757=68, srv1929547911=43, srv1692009514=33, srv1119758026=7, srv419126424=65, srv348493608=63, srv618018085=75, srv1416132389=21, srv252616210=58, srv289635978=59, srv1331104194=15, srv1817623213=36, srv1976867333=47, srv2122485981=56, srv928458748=96, srv1891815345=41, srv344601824=61, srv938063459=97, srv1554130133=23, srv224658972=57, srv1453401233=22, srv1271655892=12, srv309207915=60, srv54178412=70, srv62708963=76, srv1881342767=40, srv669538430=79, srv347255023=62, srv1078985917=4, srv1561000048=24, srv85647128=92, srv1838108622=38, srv161244321=27, srv583522121=73, srv1732054744=35, srv746202407=82, srv99878656=99, srv1668615708=32, srv835461685=89, srv1398583404=20, srv982349716=98, srv1331901251=16, srv848915681=91, srv1034923926=1, srv755976783=84, srv1046553179=2, srv2084168914=52, srv677918921=80, srv1068582366=3, srv1136645188=8, srv848576952=90, srv1388779071=19, srv824690011=87, srv1591149025=26, srv2064067865=51, srv1026068441=0, srv470128112=66, srv580749837=72, srv835131595=88, srv109210577=6, srv1942672919=44, srv1573939300=25, srv1343649209=17, srv1916240833=42, srv501724490=67, srv165898153=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:22,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:22,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:22,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-31 08:54:22,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-31 08:54:22,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-31 08:54:22,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-31 08:54:22,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-31 08:54:22,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-31 08:54:22,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-31 08:54:22,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-31 08:54:22,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01892878865063798); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027438329994042148, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01892878865063798); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0095); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043166666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:22,794 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-31 08:54:22,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012653499226588575, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01892878865063798); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027438329994042148, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01892878865063798); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0095); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043166666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-31 08:54:23,123 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations. Found a solution that moves 969 regions; Going from a computed imbalance of 0.012653499226588575 to a new imbalance of 5.729960847790505E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004639383664145491); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058218983640361864); MoveCostFunction : (multiplier=7.0, imbalance=0.10766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004639383664145491); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:23,125 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2084168914:81 , srv665256066:81 , srv1573939300:82 , srv1591149025:82 , srv669538430:82 , srv1554130133:83 , srv1838108622:83 , srv1929547911:83 , srv1978201535:83 , srv2050433505:83 , srv347255023:83 , srv54178412:83 , srv1026068441:84 , srv1068582366:84 , srv1078985917:84 , srv1136645188:84 , srv1136898638:84 , srv1193559265:84 , srv1344969402:84 , srv1416132389:84 , srv1613291777:84 , srv1632430351:84 , srv1692009514:84 , srv1730446140:84 , srv2120632631:84 , srv252616210:84 , srv348493608:84 , srv583522121:84 , srv583701446:84 , srv618018085:84 , srv848915681:84 , srv928458748:84 , srv99878656:84 , srv1292835889:85 , srv1343649209:85 , srv1388779071:85 , srv1668615708:85 , srv1878294466:85 , srv1976867333:85 , srv208509088:85 , srv289635978:85 , srv309207915:85 , srv419126424:85 , srv580749837:85 , srv824690011:85 , srv835461685:85 , srv848576952:85 , srv876085335:85 , srv919529277:85 , srv1046553179:86 , srv109210577:86 , srv1119758026:86 , srv1214560921:86 , srv1331901251:86 , srv1624236161:86 , srv165898153:86 , srv1817623213:86 , srv1827068063:86 , srv1891815345:86 , srv1916240833:86 , srv194876440:86 , srv1976837660:86 , srv2007327392:86 , srv2064067865:86 , srv344601824:86 , srv393951548:86 , srv504931757:86 , srv533074508:86 , srv62708963:86 , srv636148501:86 , srv746202407:86 , srv750263286:86 , srv85647128:86 , srv1034923926:87 , srv10905006:87 , srv1271655892:87 , srv1314861810:87 , srv1331104194:87 , srv1453401233:87 , srv1561000048:87 , srv161244321:87 , srv1732054744:87 , srv1942672919:87 , srv2121160851:87 , srv2122485981:87 , srv224658972:87 , srv470128112:87 , srv501724490:87 , srv565958855:87 , srv677918921:87 , srv755976783:87 , srv767080301:87 , srv835131595:87 , srv868891094:87 , srv938063459:87 , srv1398583404:88 , srv700800755:88 , srv807095526:88 , srv982349716:88 , srv1881342767:557 } 2023-08-31 08:54:23,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:23,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv194876440=45, srv750263286=83, srv1632430351=30, srv10905006=5, srv2007327392=49, srv393951548=64, srv2120632631=54, srv1136898638=9, srv807095526=86, srv700800755=81, srv533074508=69, srv1193559265=10, srv1730446140=34, srv1613291777=28, srv1878294466=39, srv2121160851=55, srv208509088=53, srv665256066=78, srv876085335=94, srv1344969402=18, srv1978201535=48, srv868891094=93, srv1976837660=46, srv565958855=71, srv919529277=95, srv1214560921=11, srv1827068063=37, srv583701446=74, srv1624236161=29, srv2050433505=50, srv636148501=77, srv1292835889=13, srv1314861810=14, srv767080301=85, srv504931757=68, srv1929547911=43, srv1692009514=33, srv1119758026=7, srv419126424=65, srv348493608=63, srv618018085=75, srv1416132389=21, srv252616210=58, srv289635978=59, srv1331104194=15, srv1817623213=36, srv1976867333=47, srv2122485981=56, srv928458748=96, srv1891815345=41, srv344601824=61, srv938063459=97, srv1554130133=23, srv224658972=57, srv1453401233=22, srv1271655892=12, srv309207915=60, srv54178412=70, srv62708963=76, srv1881342767=40, srv669538430=79, srv347255023=62, srv1078985917=4, srv1561000048=24, srv85647128=92, srv1838108622=38, srv161244321=27, srv583522121=73, srv1732054744=35, srv746202407=82, srv99878656=99, srv1668615708=32, srv835461685=89, srv1398583404=20, srv982349716=98, srv1331901251=16, srv848915681=91, srv1034923926=1, srv755976783=84, srv1046553179=2, srv2084168914=52, srv677918921=80, srv1068582366=3, srv1136645188=8, srv848576952=90, srv1388779071=19, srv824690011=87, srv1591149025=26, srv2064067865=51, srv1026068441=0, srv470128112=66, srv580749837=72, srv835131595=88, srv109210577=6, srv1942672919=44, srv1573939300=25, srv1343649209=17, srv1916240833=42, srv501724490=67, srv165898153=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-31 08:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-31 08:54:23,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-31 08:54:23,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-31 08:54:23,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-31 08:54:23,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-31 08:54:23,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-31 08:54:23,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-31 08:54:23,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-31 08:54:23,309 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004639383664145491); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058218983640361864); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004639383664145491); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:23,309 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-31 08:54:23,310 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.662110017585632E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004639383664145491); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058218983640361864); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004639383664145491); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-31 08:54:23,789 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 618 ms to try 5000 different iterations. Found a solution that moves 217 regions; Going from a computed imbalance of 5.662110017585632E-4 to a new imbalance of 2.695605513285324E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.02411111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:23,790 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1026068441:83 , srv1119758026:83 , srv1136645188:83 , srv1942672919:83 , srv54178412:83 , srv85647128:83 , srv982349716:83 , srv1034923926:84 , srv1046553179:84 , srv10905006:84 , srv109210577:84 , srv1214560921:84 , srv1314861810:84 , srv1331104194:84 , srv1331901251:84 , srv1343649209:84 , srv1388779071:84 , srv1398583404:84 , srv1453401233:84 , srv1554130133:84 , srv1573939300:84 , srv1591149025:84 , srv161244321:84 , srv1624236161:84 , srv165898153:84 , srv1692009514:84 , srv1732054744:84 , srv1817623213:84 , srv1838108622:84 , srv1878294466:84 , srv1976837660:84 , srv1978201535:84 , srv2050433505:84 , srv252616210:84 , srv289635978:84 , srv309207915:84 , srv344601824:84 , srv393951548:84 , srv419126424:84 , srv470128112:84 , srv504931757:84 , srv565958855:84 , srv583522121:84 , srv636148501:84 , srv669538430:84 , srv700800755:84 , srv746202407:84 , srv750263286:84 , srv755976783:84 , srv767080301:84 , srv807095526:84 , srv824690011:84 , srv835131595:84 , srv919529277:84 , srv928458748:84 , srv938063459:84 , srv1068582366:85 , srv1078985917:85 , srv1136898638:85 , srv1193559265:85 , srv1271655892:85 , srv1292835889:85 , srv1344969402:85 , srv1416132389:85 , srv1561000048:85 , srv1613291777:85 , srv1632430351:85 , srv1668615708:85 , srv1730446140:85 , srv1827068063:85 , srv1891815345:85 , srv1916240833:85 , srv1929547911:85 , srv194876440:85 , srv1976867333:85 , srv2007327392:85 , srv2064067865:85 , srv2084168914:85 , srv208509088:85 , srv2120632631:85 , srv2122485981:85 , srv224658972:85 , srv347255023:85 , srv348493608:85 , srv501724490:85 , srv580749837:85 , srv618018085:85 , srv62708963:85 , srv665256066:85 , srv677918921:85 , srv835461685:85 , srv848576952:85 , srv868891094:85 , srv876085335:85 , srv99878656:85 , srv2121160851:86 , srv533074508:86 , srv583701446:86 , srv848915681:86 , srv1881342767:644 } 2023-08-31 08:54:23,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:23,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv194876440=45, srv750263286=83, srv1632430351=30, srv10905006=5, srv2007327392=49, srv393951548=64, srv2120632631=54, srv1136898638=9, srv807095526=86, srv700800755=81, srv533074508=69, srv1193559265=10, srv1730446140=34, srv1613291777=28, srv1878294466=39, srv2121160851=55, srv208509088=53, srv665256066=78, srv876085335=94, srv1344969402=18, srv1978201535=48, srv868891094=93, srv1976837660=46, srv565958855=71, srv919529277=95, srv1214560921=11, srv1827068063=37, srv583701446=74, srv1624236161=29, srv2050433505=50, srv636148501=77, srv1292835889=13, srv1314861810=14, srv767080301=85, srv504931757=68, srv1929547911=43, srv1692009514=33, srv1119758026=7, srv419126424=65, srv348493608=63, srv618018085=75, srv1416132389=21, srv252616210=58, srv289635978=59, srv1331104194=15, srv1817623213=36, srv1976867333=47, srv2122485981=56, srv928458748=96, srv1891815345=41, srv344601824=61, srv938063459=97, srv1554130133=23, srv224658972=57, srv1453401233=22, srv1271655892=12, srv309207915=60, srv54178412=70, srv62708963=76, srv1881342767=40, srv669538430=79, srv347255023=62, srv1078985917=4, srv1561000048=24, srv85647128=92, srv1838108622=38, srv161244321=27, srv583522121=73, srv1732054744=35, srv746202407=82, srv99878656=99, srv1668615708=32, srv835461685=89, srv1398583404=20, srv982349716=98, srv1331901251=16, srv848915681=91, srv1034923926=1, srv755976783=84, srv1046553179=2, srv2084168914=52, srv677918921=80, srv1068582366=3, srv1136645188=8, srv848576952=90, srv1388779071=19, srv824690011=87, srv1591149025=26, srv2064067865=51, srv1026068441=0, srv470128112=66, srv580749837=72, srv835131595=88, srv109210577=6, srv1942672919=44, srv1573939300=25, srv1343649209=17, srv1916240833=42, srv501724490=67, srv165898153=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:23,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:23,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:23,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-31 08:54:23,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-31 08:54:23,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-31 08:54:23,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-31 08:54:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-31 08:54:23,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-31 08:54:23,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-31 08:54:23,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-31 08:54:23,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-31 08:54:23,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-31 08:54:23,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-31 08:54:23,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-31 08:54:23,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-31 08:54:23,969 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:23,969 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-31 08:54:23,969 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.5436588656645044E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-31 08:54:24,279 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 443 ms to try 5000 different iterations. Found a solution that moves 25 regions; Going from a computed imbalance of 2.5436588656645044E-5 to a new imbalance of 6.5730394137683395E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.002777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:24,280 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1827068063:83 , srv1891815345:83 , srv194876440:83 , srv1976867333:83 , srv2007327392:83 , srv208509088:83 , srv2121160851:83 , srv224658972:83 , srv501724490:83 , srv533074508:83 , srv583701446:83 , srv848915681:83 , srv1034923926:84 , srv1046553179:84 , srv1068582366:84 , srv1078985917:84 , srv10905006:84 , srv109210577:84 , srv1119758026:84 , srv1136898638:84 , srv1193559265:84 , srv1214560921:84 , srv1271655892:84 , srv1292835889:84 , srv1314861810:84 , srv1331104194:84 , srv1331901251:84 , srv1343649209:84 , srv1344969402:84 , srv1388779071:84 , srv1398583404:84 , srv1416132389:84 , srv1453401233:84 , srv1561000048:84 , srv1573939300:84 , srv1591149025:84 , srv161244321:84 , srv1613291777:84 , srv1624236161:84 , srv1632430351:84 , srv1668615708:84 , srv1692009514:84 , srv1730446140:84 , srv1732054744:84 , srv1817623213:84 , srv1838108622:84 , srv1878294466:84 , srv1916240833:84 , srv1929547911:84 , srv1976837660:84 , srv2064067865:84 , srv2084168914:84 , srv2120632631:84 , srv2122485981:84 , srv252616210:84 , srv309207915:84 , srv344601824:84 , srv347255023:84 , srv348493608:84 , srv419126424:84 , srv470128112:84 , srv504931757:84 , srv565958855:84 , srv580749837:84 , srv583522121:84 , srv618018085:84 , srv62708963:84 , srv636148501:84 , srv665256066:84 , srv669538430:84 , srv677918921:84 , srv700800755:84 , srv746202407:84 , srv750263286:84 , srv755976783:84 , srv767080301:84 , srv807095526:84 , srv824690011:84 , srv835461685:84 , srv848576952:84 , srv85647128:84 , srv868891094:84 , srv876085335:84 , srv928458748:84 , srv938063459:84 , srv99878656:84 , srv1026068441:85 , srv1136645188:85 , srv1554130133:85 , srv165898153:85 , srv1942672919:85 , srv1978201535:85 , srv2050433505:85 , srv289635978:85 , srv393951548:85 , srv54178412:85 , srv835131595:85 , srv919529277:85 , srv982349716:85 , srv1881342767:683 } 2023-08-31 08:54:24,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:24,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv194876440=45, srv750263286=83, srv1632430351=30, srv10905006=5, srv2007327392=49, srv393951548=64, srv2120632631=54, srv1136898638=9, srv807095526=86, srv700800755=81, srv533074508=69, srv1193559265=10, srv1730446140=34, srv1613291777=28, srv1878294466=39, srv2121160851=55, srv208509088=53, srv665256066=78, srv876085335=94, srv1344969402=18, srv1978201535=48, srv868891094=93, srv1976837660=46, srv565958855=71, srv919529277=95, srv1214560921=11, srv1827068063=37, srv583701446=74, srv1624236161=29, srv2050433505=50, srv636148501=77, srv1292835889=13, srv1314861810=14, srv767080301=85, srv504931757=68, srv1929547911=43, srv1692009514=33, srv1119758026=7, srv419126424=65, srv348493608=63, srv618018085=75, srv1416132389=21, srv252616210=58, srv289635978=59, srv1331104194=15, srv1817623213=36, srv1976867333=47, srv2122485981=56, srv928458748=96, srv1891815345=41, srv344601824=61, srv938063459=97, srv1554130133=23, srv224658972=57, srv1453401233=22, srv1271655892=12, srv309207915=60, srv54178412=70, srv62708963=76, srv1881342767=40, srv669538430=79, srv347255023=62, srv1078985917=4, srv1561000048=24, srv85647128=92, srv1838108622=38, srv161244321=27, srv583522121=73, srv1732054744=35, srv746202407=82, srv99878656=99, srv1668615708=32, srv835461685=89, srv1398583404=20, srv982349716=98, srv1331901251=16, srv848915681=91, srv1034923926=1, srv755976783=84, srv1046553179=2, srv2084168914=52, srv677918921=80, srv1068582366=3, srv1136645188=8, srv848576952=90, srv1388779071=19, srv824690011=87, srv1591149025=26, srv2064067865=51, srv1026068441=0, srv470128112=66, srv580749837=72, srv835131595=88, srv109210577=6, srv1942672919=44, srv1573939300=25, srv1343649209=17, srv1916240833=42, srv501724490=67, srv165898153=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:24,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:24,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:24,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-31 08:54:24,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-31 08:54:24,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-31 08:54:24,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-31 08:54:24,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-31 08:54:24,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-31 08:54:24,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-31 08:54:24,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-31 08:54:24,421 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:24,422 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1827068063:83 , srv1891815345:83 , srv194876440:83 , srv1976867333:83 , srv2007327392:83 , srv208509088:83 , srv2121160851:83 , srv224658972:83 , srv501724490:83 , srv533074508:83 , srv583701446:83 , srv848915681:83 , srv1034923926:84 , srv1046553179:84 , srv1068582366:84 , srv1078985917:84 , srv10905006:84 , srv109210577:84 , srv1119758026:84 , srv1136898638:84 , srv1193559265:84 , srv1214560921:84 , srv1271655892:84 , srv1292835889:84 , srv1314861810:84 , srv1331104194:84 , srv1331901251:84 , srv1343649209:84 , srv1344969402:84 , srv1388779071:84 , srv1398583404:84 , srv1416132389:84 , srv1453401233:84 , srv1561000048:84 , srv1573939300:84 , srv1591149025:84 , srv161244321:84 , srv1613291777:84 , srv1624236161:84 , srv1632430351:84 , srv1668615708:84 , srv1692009514:84 , srv1730446140:84 , srv1732054744:84 , srv1817623213:84 , srv1838108622:84 , srv1878294466:84 , srv1916240833:84 , srv1929547911:84 , srv1976837660:84 , srv2064067865:84 , srv2084168914:84 , srv2120632631:84 , srv2122485981:84 , srv252616210:84 , srv309207915:84 , srv344601824:84 , srv347255023:84 , srv348493608:84 , srv419126424:84 , srv470128112:84 , srv504931757:84 , srv565958855:84 , srv580749837:84 , srv583522121:84 , srv618018085:84 , srv62708963:84 , srv636148501:84 , srv665256066:84 , srv669538430:84 , srv677918921:84 , srv700800755:84 , srv746202407:84 , srv750263286:84 , srv755976783:84 , srv767080301:84 , srv807095526:84 , srv824690011:84 , srv835461685:84 , srv848576952:84 , srv85647128:84 , srv868891094:84 , srv876085335:84 , srv928458748:84 , srv938063459:84 , srv99878656:84 , srv1026068441:85 , srv1136645188:85 , srv1554130133:85 , srv165898153:85 , srv1942672919:85 , srv1978201535:85 , srv2050433505:85 , srv289635978:85 , srv393951548:85 , srv54178412:85 , srv835131595:85 , srv919529277:85 , srv982349716:85 , srv1881342767:683 } 2023-08-31 08:54:24,446 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=398 (was 354) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9883 (was 10775) 2023-08-31 08:54:24,454 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=398, ProcessCount=167, AvailableMemoryMB=9883 2023-08-31 08:54:24,455 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-31 08:54:24,455 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-31 08:54:24,467 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1053429055:84 , srv1058196871:84 , srv1102720163:84 , srv1349564691:84 , srv1380720703:84 , srv1403327830:84 , srv1482771548:84 , srv1641448133:84 , srv1876757520:84 , srv1927508066:84 , srv1927784217:84 , srv1930612059:84 , srv2000052008:84 , srv2010544786:84 , srv2013928620:84 , srv202832874:84 , srv2043249676:84 , srv2088697257:84 , srv240719513:84 , srv294226261:84 , srv303492949:84 , srv506001474:84 , srv536531955:84 , srv617682825:84 , srv686592059:84 , srv751497093:84 , srv770676776:84 , srv893324407:84 , srv942577203:84 , srv2041797609:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-31 08:54:24,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:54:24,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2043249676=17, srv617682825=24, srv294226261=20, srv240719513=19, srv1380720703=4, srv2041797609=16, srv2013928620=14, srv536531955=23, srv1349564691=3, srv2010544786=13, srv1876757520=8, srv1927784217=10, srv686592059=25, srv1641448133=7, srv1482771548=6, srv1102720163=2, srv2000052008=12, srv303492949=21, srv770676776=27, srv942577203=29, srv2088697257=18, srv1058196871=1, srv751497093=26, srv1053429055=0, srv506001474=22, srv1403327830=5, srv1927508066=9, srv202832874=15, srv893324407=28, srv1930612059=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:54:24,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:54:24,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:54:24,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:54:24,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:54:24,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:54:24,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:54:24,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:54:24,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-31 08:54:24,510 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0647625678745953, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:54:24,510 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912397108000961, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0647625678745953, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-31 08:56:24,489 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5628384 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912397108000961 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:56:24,497 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1053429055:90 , srv1058196871:90 , srv1102720163:90 , srv1349564691:90 , srv1380720703:90 , srv1403327830:90 , srv1482771548:90 , srv1641448133:90 , srv1876757520:90 , srv1927508066:90 , srv1927784217:90 , srv1930612059:90 , srv2000052008:90 , srv2010544786:90 , srv2013928620:90 , srv202832874:90 , srv2041797609:90 , srv2043249676:90 , srv2088697257:90 , srv240719513:90 , srv294226261:90 , srv303492949:90 , srv506001474:90 , srv536531955:90 , srv617682825:90 , srv686592059:90 , srv751497093:90 , srv770676776:90 , srv893324407:90 , srv942577203:90 } 2023-08-31 08:56:24,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-31 08:56:24,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2043249676=17, srv617682825=24, srv294226261=20, srv240719513=19, srv1380720703=4, srv2041797609=16, srv2013928620=14, srv536531955=23, srv1349564691=3, srv2010544786=13, srv1876757520=8, srv1927784217=10, srv686592059=25, srv1641448133=7, srv1482771548=6, srv1102720163=2, srv2000052008=12, srv303492949=21, srv770676776=27, srv942577203=29, srv2088697257=18, srv1058196871=1, srv751497093=26, srv1053429055=0, srv506001474=22, srv1403327830=5, srv1927508066=9, srv202832874=15, srv893324407=28, srv1930612059=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-31 08:56:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-31 08:56:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-31 08:56:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-31 08:56:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-31 08:56:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-31 08:56:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-31 08:56:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-31 08:56:24,517 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-31 08:56:24,518 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1053429055:90 , srv1058196871:90 , srv1102720163:90 , srv1349564691:90 , srv1380720703:90 , srv1403327830:90 , srv1482771548:90 , srv1641448133:90 , srv1876757520:90 , srv1927508066:90 , srv1927784217:90 , srv1930612059:90 , srv2000052008:90 , srv2010544786:90 , srv2013928620:90 , srv202832874:90 , srv2041797609:90 , srv2043249676:90 , srv2088697257:90 , srv240719513:90 , srv294226261:90 , srv303492949:90 , srv506001474:90 , srv536531955:90 , srv617682825:90 , srv686592059:90 , srv751497093:90 , srv770676776:90 , srv893324407:90 , srv942577203:90 } 2023-08-31 08:56:24,530 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 398), ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=9687 (was 9883)