2023-09-23 02:53:15,845 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-23 02:53:16,175 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-23 02:53:16,296 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-23 02:53:16,361 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 02:53:16,368 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-23 02:53:16,385 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=362, ProcessCount=172, AvailableMemoryMB=3264 2023-09-23 02:53:16,387 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 02:53:16,388 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-23 02:53:16,540 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023783130:84 , srv102682240:84 , srv1027598662:84 , srv106770311:84 , srv1080874717:84 , srv1098860636:84 , srv1122339761:84 , srv112996079:84 , srv1138406613:84 , srv119568927:84 , srv1210017256:84 , srv1210816908:84 , srv1219020934:84 , srv1235655751:84 , srv1268196995:84 , srv1269508605:84 , srv129902853:84 , srv1325649140:84 , srv1338934828:84 , srv1339098630:84 , srv1350141397:84 , srv1374470165:84 , srv1413123205:84 , srv142472650:84 , srv1462158755:84 , srv1466750648:84 , srv1481217451:84 , srv1530749803:84 , srv1560628284:84 , srv1576311808:84 , srv1589287317:84 , srv1617160447:84 , srv1623853093:84 , srv1635447070:84 , srv164363376:84 , srv1657810459:84 , srv1698131216:84 , srv1748966733:84 , srv1761470354:84 , srv1761733245:84 , srv1764843534:84 , srv1775864274:84 , srv1779837539:84 , srv184125165:84 , srv1852979621:84 , srv1854353846:84 , srv1859838797:84 , srv1874874846:84 , srv1908491504:84 , srv191186670:84 , srv1918019972:84 , srv1921934901:84 , srv1953872553:84 , srv195658441:84 , srv1992013899:84 , srv1994493517:84 , srv1995793815:84 , srv2019405391:84 , srv2019880009:84 , srv2042115383:84 , srv2051714882:84 , srv2064451594:84 , srv2134688338:84 , srv2138219327:84 , srv214141867:84 , srv2144291918:84 , srv236847643:84 , srv260265510:84 , srv334228228:84 , srv373944622:84 , srv376316467:84 , srv379144354:84 , srv39486154:84 , srv402965939:84 , srv412430804:84 , srv415484178:84 , srv424686333:84 , srv435721843:84 , srv436528360:84 , srv490421725:84 , srv511621247:84 , srv557285929:84 , srv567315261:84 , srv587423552:84 , srv590727050:84 , srv60809922:84 , srv623802888:84 , srv639428532:84 , srv649161505:84 , srv661057650:84 , srv694268881:84 , srv711008607:84 , srv717192263:84 , srv724416326:84 , srv756442259:84 , srv789940722:84 , srv806333866:84 , srv858146254:84 , srv917469945:84 , srv2099251233:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-23 02:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv102682240=1, srv2019880009=58, srv2042115383=59, srv1992013899=54, srv424686333=77, srv184125165=43, srv694268881=91, srv119568927=9, srv1852979621=44, srv1854353846=45, srv236847643=67, srv1859838797=46, srv1918019972=50, srv490421725=80, srv1635447070=33, srv379144354=72, srv590727050=85, srv2019405391=57, srv587423552=84, srv649161505=89, srv1617160447=31, srv1761470354=38, srv1764843534=40, srv1995793815=56, srv639428532=88, srv1122339761=6, srv1874874846=47, srv1466750648=25, srv1481217451=26, srv1775864274=41, srv1589287317=30, srv1219020934=12, srv1235655751=13, srv373944622=70, srv623802888=87, srv557285929=82, srv436528360=79, srv334228228=69, srv1023783130=0, srv1921934901=51, srv2064451594=61, srv1268196995=14, srv1138406613=8, srv1462158755=24, srv789940722=96, srv717192263=93, srv756442259=95, srv1098860636=5, srv1325649140=17, srv1350141397=20, srv60809922=86, srv195658441=53, srv1698131216=36, srv1560628284=28, srv214141867=65, srv858146254=98, srv1413123205=22, srv164363376=34, srv511621247=81, srv2051714882=60, srv412430804=75, srv2138219327=64, srv1338934828=18, srv1576311808=29, srv112996079=7, srv1761733245=39, srv1908491504=48, srv402965939=74, srv2134688338=63, srv1210017256=10, srv260265510=68, srv661057650=90, srv1269508605=15, srv106770311=3, srv1779837539=42, srv711008607=92, srv806333866=97, srv1530749803=27, srv129902853=16, srv1748966733=37, srv1953872553=52, srv1994493517=55, srv2099251233=62, srv1657810459=35, srv1210816908=11, srv1080874717=4, srv724416326=94, srv1623853093=32, srv1027598662=2, srv142472650=23, srv435721843=78, srv567315261=83, srv1374470165=21, srv191186670=49, srv2144291918=66, srv917469945=99, srv415484178=76, srv1339098630=19, srv376316467=71, srv39486154=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:16,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:16,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:16,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:16,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:16,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:16,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:16,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:16,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:16,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:16,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:16,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:16,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:16,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:16,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:16,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:16,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:16,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:16,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:16,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 02:53:16,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 02:53:16,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 02:53:16,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 02:53:16,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 02:53:16,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 02:53:16,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 02:53:16,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 02:53:16,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 02:53:16,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 02:53:16,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 02:53:16,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 02:53:16,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 02:53:16,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:16,840 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 02:53:16,840 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 02:53:17,187 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations. Found a solution that moves 4623 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18883679322097957. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03705050260266537, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05901359009530897, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5136666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03705050260266537, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18416666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2505833333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:17,210 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1698131216:81 , srv191186670:81 , srv1764843534:82 , srv2064451594:82 , srv379144354:82 , srv587423552:82 , srv1635447070:83 , srv2144291918:83 , srv373944622:83 , srv60809922:83 , srv858146254:83 , srv1219020934:84 , srv1269508605:84 , srv1339098630:84 , srv1374470165:84 , srv1413123205:84 , srv1466750648:84 , srv1761470354:84 , srv1854353846:84 , srv195658441:84 , srv2138219327:84 , srv260265510:84 , srv694268881:84 , srv1138406613:85 , srv1210816908:85 , srv1325649140:85 , srv1617160447:85 , srv1779837539:85 , srv1918019972:85 , srv214141867:85 , srv334228228:85 , srv511621247:85 , srv567315261:85 , srv623802888:85 , srv106770311:86 , srv1098860636:86 , srv1338934828:86 , srv1462158755:86 , srv1623853093:86 , srv1775864274:86 , srv1859838797:86 , srv2019405391:86 , srv412430804:86 , srv424686333:86 , srv435721843:86 , srv661057650:86 , srv724416326:86 , srv756442259:86 , srv917469945:86 , srv1023783130:87 , srv1235655751:87 , srv1481217451:87 , srv184125165:87 , srv1852979621:87 , srv1953872553:87 , srv1994493517:87 , srv2042115383:87 , srv2134688338:87 , srv39486154:87 , srv415484178:87 , srv436528360:87 , srv717192263:87 , srv789940722:87 , srv1027598662:88 , srv112996079:88 , srv1210017256:88 , srv1530749803:88 , srv1560628284:88 , srv1589287317:88 , srv1761733245:88 , srv1921934901:88 , srv1995793815:88 , srv236847643:88 , srv649161505:88 , srv711008607:88 , srv1122339761:89 , srv129902853:89 , srv1350141397:89 , srv164363376:89 , srv1657810459:89 , srv1748966733:89 , srv1908491504:89 , srv2019880009:89 , srv402965939:89 , srv1080874717:90 , srv1268196995:90 , srv1576311808:90 , srv490421725:90 , srv590727050:90 , srv806333866:90 , srv119568927:91 , srv1874874846:91 , srv2051714882:91 , srv639428532:91 , srv102682240:92 , srv142472650:92 , srv376316467:93 , srv557285929:93 , srv1992013899:94 , srv2099251233:419 } 2023-09-23 02:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv102682240=1, srv2019880009=58, srv2042115383=59, srv1992013899=54, srv424686333=77, srv184125165=43, srv694268881=91, srv119568927=9, srv1852979621=44, srv1854353846=45, srv236847643=67, srv1859838797=46, srv1918019972=50, srv490421725=80, srv1635447070=33, srv379144354=72, srv590727050=85, srv2019405391=57, srv587423552=84, srv649161505=89, srv1617160447=31, srv1761470354=38, srv1764843534=40, srv1995793815=56, srv639428532=88, srv1122339761=6, srv1874874846=47, srv1466750648=25, srv1481217451=26, srv1775864274=41, srv1589287317=30, srv1219020934=12, srv1235655751=13, srv373944622=70, srv623802888=87, srv557285929=82, srv436528360=79, srv334228228=69, srv1023783130=0, srv1921934901=51, srv2064451594=61, srv1268196995=14, srv1138406613=8, srv1462158755=24, srv789940722=96, srv717192263=93, srv756442259=95, srv1098860636=5, srv1325649140=17, srv1350141397=20, srv60809922=86, srv195658441=53, srv1698131216=36, srv1560628284=28, srv214141867=65, srv858146254=98, srv1413123205=22, srv164363376=34, srv511621247=81, srv2051714882=60, srv412430804=75, srv2138219327=64, srv1338934828=18, srv1576311808=29, srv112996079=7, srv1761733245=39, srv1908491504=48, srv402965939=74, srv2134688338=63, srv1210017256=10, srv260265510=68, srv661057650=90, srv1269508605=15, srv106770311=3, srv1779837539=42, srv711008607=92, srv806333866=97, srv1530749803=27, srv129902853=16, srv1748966733=37, srv1953872553=52, srv1994493517=55, srv2099251233=62, srv1657810459=35, srv1210816908=11, srv1080874717=4, srv724416326=94, srv1623853093=32, srv1027598662=2, srv142472650=23, srv435721843=78, srv567315261=83, srv1374470165=21, srv191186670=49, srv2144291918=66, srv917469945=99, srv415484178=76, srv1339098630=19, srv376316467=71, srv39486154=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:17,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:17,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:17,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:17,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:17,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:17,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 02:53:17,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 02:53:17,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 02:53:17,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 02:53:17,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 02:53:17,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 02:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 02:53:17,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 02:53:17,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03705050260266537, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05901359009530897, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03705050260266537, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18416666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2505833333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:17,415 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 02:53:17,415 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18880442228310163, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03705050260266537, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05901359009530897, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03705050260266537, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18416666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2505833333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 02:53:17,690 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 418 ms to try 5000 different iterations. Found a solution that moves 2671 regions; Going from a computed imbalance of 0.18880442228310163 to a new imbalance of 0.016917113406365165. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02126501875313993); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03629595272289805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29677777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02126501875313993); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04808333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:17,701 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1992013899:78 , srv557285929:78 , srv102682240:80 , srv142472650:80 , srv806333866:80 , srv1268196995:81 , srv1589287317:81 , srv1657810459:81 , srv376316467:81 , srv119568927:82 , srv1530749803:82 , srv1576311808:82 , srv1874874846:82 , srv1908491504:82 , srv1921934901:82 , srv2019880009:82 , srv2051714882:82 , srv490421725:82 , srv1122339761:83 , srv1210017256:83 , srv164363376:83 , srv1748966733:83 , srv236847643:83 , srv639428532:83 , srv711008607:83 , srv1023783130:84 , srv1027598662:84 , srv1080874717:84 , srv1098860636:84 , srv129902853:84 , srv1350141397:84 , srv1560628284:84 , srv1761733245:84 , srv1995793815:84 , srv2134688338:84 , srv402965939:84 , srv590727050:84 , srv106770311:85 , srv112996079:85 , srv1779837539:85 , srv2019405391:85 , srv412430804:85 , srv424686333:85 , srv436528360:85 , srv1338934828:86 , srv1339098630:86 , srv1466750648:86 , srv1852979621:86 , srv1859838797:86 , srv195658441:86 , srv2042115383:86 , srv214141867:86 , srv334228228:86 , srv39486154:86 , srv435721843:86 , srv567315261:86 , srv623802888:86 , srv649161505:86 , srv661057650:86 , srv789940722:86 , srv1138406613:87 , srv1235655751:87 , srv1325649140:87 , srv1413123205:87 , srv1481217451:87 , srv1761470354:87 , srv1775864274:87 , srv1918019972:87 , srv1953872553:87 , srv1994493517:87 , srv415484178:87 , srv511621247:87 , srv717192263:87 , srv724416326:87 , srv858146254:87 , srv917469945:87 , srv1210816908:88 , srv1374470165:88 , srv1462158755:88 , srv1623853093:88 , srv1764843534:88 , srv184125165:88 , srv1854353846:88 , srv260265510:88 , srv694268881:88 , srv1219020934:89 , srv1269508605:89 , srv1617160447:89 , srv1635447070:89 , srv191186670:89 , srv2138219327:89 , srv2144291918:89 , srv587423552:89 , srv60809922:89 , srv756442259:89 , srv1698131216:90 , srv373944622:90 , srv379144354:91 , srv2064451594:93 , srv2099251233:544 } 2023-09-23 02:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv102682240=1, srv2019880009=58, srv2042115383=59, srv1992013899=54, srv424686333=77, srv184125165=43, srv694268881=91, srv119568927=9, srv1852979621=44, srv1854353846=45, srv236847643=67, srv1859838797=46, srv1918019972=50, srv490421725=80, srv1635447070=33, srv379144354=72, srv590727050=85, srv2019405391=57, srv587423552=84, srv649161505=89, srv1617160447=31, srv1761470354=38, srv1764843534=40, srv1995793815=56, srv639428532=88, srv1122339761=6, srv1874874846=47, srv1466750648=25, srv1481217451=26, srv1775864274=41, srv1589287317=30, srv1219020934=12, srv1235655751=13, srv373944622=70, srv623802888=87, srv557285929=82, srv436528360=79, srv334228228=69, srv1023783130=0, srv1921934901=51, srv2064451594=61, srv1268196995=14, srv1138406613=8, srv1462158755=24, srv789940722=96, srv717192263=93, srv756442259=95, srv1098860636=5, srv1325649140=17, srv1350141397=20, srv60809922=86, srv195658441=53, srv1698131216=36, srv1560628284=28, srv214141867=65, srv858146254=98, srv1413123205=22, srv164363376=34, srv511621247=81, srv2051714882=60, srv412430804=75, srv2138219327=64, srv1338934828=18, srv1576311808=29, srv112996079=7, srv1761733245=39, srv1908491504=48, srv402965939=74, srv2134688338=63, srv1210017256=10, srv260265510=68, srv661057650=90, srv1269508605=15, srv106770311=3, srv1779837539=42, srv711008607=92, srv806333866=97, srv1530749803=27, srv129902853=16, srv1748966733=37, srv1953872553=52, srv1994493517=55, srv2099251233=62, srv1657810459=35, srv1210816908=11, srv1080874717=4, srv724416326=94, srv1623853093=32, srv1027598662=2, srv142472650=23, srv435721843=78, srv567315261=83, srv1374470165=21, srv191186670=49, srv2144291918=66, srv917469945=99, srv415484178=76, srv1339098630=19, srv376316467=71, srv39486154=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:17,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:17,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:17,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:17,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:17,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:17,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 02:53:17,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 02:53:17,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 02:53:17,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 02:53:17,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 02:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 02:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 02:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 02:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 02:53:17,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 02:53:17,914 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02126501875313993); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03629595272289805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02126501875313993); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04808333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:17,915 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 02:53:17,915 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01689841066462345, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02126501875313993); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03629595272289805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02126501875313993); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04808333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 02:53:18,263 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 476 ms to try 5000 different iterations. Found a solution that moves 1059 regions; Going from a computed imbalance of 0.01689841066462345 to a new imbalance of 6.395293250225181E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009137925557897891); MoveCostFunction : (multiplier=7.0, imbalance=0.11766666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:18,266 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv184125165:82 , srv1235655751:83 , srv1462158755:83 , srv1617160447:83 , srv1953872553:83 , srv2064451594:83 , srv376316467:83 , srv379144354:83 , srv415484178:83 , srv590727050:83 , srv639428532:83 , srv649161505:83 , srv717192263:83 , srv1080874717:84 , srv119568927:84 , srv1219020934:84 , srv129902853:84 , srv1350141397:84 , srv1374470165:84 , srv1481217451:84 , srv1623853093:84 , srv1852979621:84 , srv1994493517:84 , srv2042115383:84 , srv2051714882:84 , srv2138219327:84 , srv373944622:84 , srv402965939:84 , srv435721843:84 , srv756442259:84 , srv917469945:84 , srv1027598662:85 , srv1138406613:85 , srv1210816908:85 , srv1269508605:85 , srv1325649140:85 , srv1775864274:85 , srv1859838797:85 , srv1874874846:85 , srv1995793815:85 , srv2144291918:85 , srv39486154:85 , srv511621247:85 , srv724416326:85 , srv1023783130:86 , srv102682240:86 , srv1122339761:86 , srv112996079:86 , srv1268196995:86 , srv1338934828:86 , srv1413123205:86 , srv142472650:86 , srv1560628284:86 , srv1576311808:86 , srv1635447070:86 , srv164363376:86 , srv1698131216:86 , srv1748966733:86 , srv1761470354:86 , srv1761733245:86 , srv1854353846:86 , srv1908491504:86 , srv1918019972:86 , srv1992013899:86 , srv2019880009:86 , srv214141867:86 , srv236847643:86 , srv260265510:86 , srv436528360:86 , srv490421725:86 , srv557285929:86 , srv567315261:86 , srv60809922:86 , srv661057650:86 , srv711008607:86 , srv789940722:86 , srv106770311:87 , srv1210017256:87 , srv1339098630:87 , srv1466750648:87 , srv1779837539:87 , srv191186670:87 , srv1921934901:87 , srv195658441:87 , srv2019405391:87 , srv2134688338:87 , srv334228228:87 , srv412430804:87 , srv424686333:87 , srv587423552:87 , srv623802888:87 , srv694268881:87 , srv806333866:87 , srv858146254:87 , srv1098860636:88 , srv1530749803:88 , srv1589287317:88 , srv1657810459:88 , srv1764843534:88 , srv2099251233:547 } 2023-09-23 02:53:18,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:18,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv102682240=1, srv2019880009=58, srv2042115383=59, srv1992013899=54, srv424686333=77, srv184125165=43, srv694268881=91, srv119568927=9, srv1852979621=44, srv1854353846=45, srv236847643=67, srv1859838797=46, srv1918019972=50, srv490421725=80, srv1635447070=33, srv379144354=72, srv590727050=85, srv2019405391=57, srv587423552=84, srv649161505=89, srv1617160447=31, srv1761470354=38, srv1764843534=40, srv1995793815=56, srv639428532=88, srv1122339761=6, srv1874874846=47, srv1466750648=25, srv1481217451=26, srv1775864274=41, srv1589287317=30, srv1219020934=12, srv1235655751=13, srv373944622=70, srv623802888=87, srv557285929=82, srv436528360=79, srv334228228=69, srv1023783130=0, srv1921934901=51, srv2064451594=61, srv1268196995=14, srv1138406613=8, srv1462158755=24, srv789940722=96, srv717192263=93, srv756442259=95, srv1098860636=5, srv1325649140=17, srv1350141397=20, srv60809922=86, srv195658441=53, srv1698131216=36, srv1560628284=28, srv214141867=65, srv858146254=98, srv1413123205=22, srv164363376=34, srv511621247=81, srv2051714882=60, srv412430804=75, srv2138219327=64, srv1338934828=18, srv1576311808=29, srv112996079=7, srv1761733245=39, srv1908491504=48, srv402965939=74, srv2134688338=63, srv1210017256=10, srv260265510=68, srv661057650=90, srv1269508605=15, srv106770311=3, srv1779837539=42, srv711008607=92, srv806333866=97, srv1530749803=27, srv129902853=16, srv1748966733=37, srv1953872553=52, srv1994493517=55, srv2099251233=62, srv1657810459=35, srv1210816908=11, srv1080874717=4, srv724416326=94, srv1623853093=32, srv1027598662=2, srv142472650=23, srv435721843=78, srv567315261=83, srv1374470165=21, srv191186670=49, srv2144291918=66, srv917469945=99, srv415484178=76, srv1339098630=19, srv376316467=71, srv39486154=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:18,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 02:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 02:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 02:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 02:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 02:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 02:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 02:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 02:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 02:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 02:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 02:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 02:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 02:53:18,466 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009137925557897891); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:18,466 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 02:53:18,467 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.321140485326357E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009137925557897891); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 02:53:18,917 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 596 ms to try 5000 different iterations. Found a solution that moves 287 regions; Going from a computed imbalance of 6.321140485326357E-4 to a new imbalance of 5.591939247290512E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0030336681651881225); MoveCostFunction : (multiplier=7.0, imbalance=0.03188888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:18,919 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1122339761:83 , srv112996079:83 , srv164363376:83 , srv756442259:83 , srv789940722:83 , srv102682240:84 , srv1027598662:84 , srv106770311:84 , srv1098860636:84 , srv1210017256:84 , srv1268196995:84 , srv1338934828:84 , srv142472650:84 , srv1481217451:84 , srv1530749803:84 , srv1560628284:84 , srv1576311808:84 , srv1623853093:84 , srv1657810459:84 , srv1748966733:84 , srv1761733245:84 , srv1764843534:84 , srv1775864274:84 , srv1854353846:84 , srv1859838797:84 , srv1874874846:84 , srv191186670:84 , srv1918019972:84 , srv1953872553:84 , srv195658441:84 , srv1992013899:84 , srv1994493517:84 , srv2019405391:84 , srv2019880009:84 , srv2144291918:84 , srv260265510:84 , srv334228228:84 , srv39486154:84 , srv412430804:84 , srv424686333:84 , srv490421725:84 , srv567315261:84 , srv587423552:84 , srv60809922:84 , srv661057650:84 , srv694268881:84 , srv711008607:84 , srv724416326:84 , srv1023783130:85 , srv1080874717:85 , srv1138406613:85 , srv1210816908:85 , srv1219020934:85 , srv1235655751:85 , srv1269508605:85 , srv129902853:85 , srv1339098630:85 , srv1350141397:85 , srv1413123205:85 , srv1466750648:85 , srv1589287317:85 , srv1617160447:85 , srv1635447070:85 , srv1698131216:85 , srv1761470354:85 , srv1779837539:85 , srv184125165:85 , srv1852979621:85 , srv1908491504:85 , srv1921934901:85 , srv1995793815:85 , srv2042115383:85 , srv2051714882:85 , srv2064451594:85 , srv2134688338:85 , srv2138219327:85 , srv214141867:85 , srv236847643:85 , srv373944622:85 , srv376316467:85 , srv402965939:85 , srv415484178:85 , srv435721843:85 , srv436528360:85 , srv511621247:85 , srv557285929:85 , srv590727050:85 , srv623802888:85 , srv639428532:85 , srv649161505:85 , srv717192263:85 , srv806333866:85 , srv858146254:85 , srv119568927:86 , srv1325649140:86 , srv1374470165:86 , srv1462158755:86 , srv379144354:86 , srv917469945:86 , srv2099251233:632 } 2023-09-23 02:53:18,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:18,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv102682240=1, srv2019880009=58, srv2042115383=59, srv1992013899=54, srv424686333=77, srv184125165=43, srv694268881=91, srv119568927=9, srv1852979621=44, srv1854353846=45, srv236847643=67, srv1859838797=46, srv1918019972=50, srv490421725=80, srv1635447070=33, srv379144354=72, srv590727050=85, srv2019405391=57, srv587423552=84, srv649161505=89, srv1617160447=31, srv1761470354=38, srv1764843534=40, srv1995793815=56, srv639428532=88, srv1122339761=6, srv1874874846=47, srv1466750648=25, srv1481217451=26, srv1775864274=41, srv1589287317=30, srv1219020934=12, srv1235655751=13, srv373944622=70, srv623802888=87, srv557285929=82, srv436528360=79, srv334228228=69, srv1023783130=0, srv1921934901=51, srv2064451594=61, srv1268196995=14, srv1138406613=8, srv1462158755=24, srv789940722=96, srv717192263=93, srv756442259=95, srv1098860636=5, srv1325649140=17, srv1350141397=20, srv60809922=86, srv195658441=53, srv1698131216=36, srv1560628284=28, srv214141867=65, srv858146254=98, srv1413123205=22, srv164363376=34, srv511621247=81, srv2051714882=60, srv412430804=75, srv2138219327=64, srv1338934828=18, srv1576311808=29, srv112996079=7, srv1761733245=39, srv1908491504=48, srv402965939=74, srv2134688338=63, srv1210017256=10, srv260265510=68, srv661057650=90, srv1269508605=15, srv106770311=3, srv1779837539=42, srv711008607=92, srv806333866=97, srv1530749803=27, srv129902853=16, srv1748966733=37, srv1953872553=52, srv1994493517=55, srv2099251233=62, srv1657810459=35, srv1210816908=11, srv1080874717=4, srv724416326=94, srv1623853093=32, srv1027598662=2, srv142472650=23, srv435721843=78, srv567315261=83, srv1374470165=21, srv191186670=49, srv2144291918=66, srv917469945=99, srv415484178=76, srv1339098630=19, srv376316467=71, srv39486154=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:18,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 02:53:18,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 02:53:18,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 02:53:18,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 02:53:18,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 02:53:18,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 02:53:19,051 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0030336681651881225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:19,051 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-23 02:53:19,051 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.390977552050073E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0030336681651881225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-23 02:53:19,294 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 338 ms to try 5000 different iterations. Found a solution that moves 62 regions; Going from a computed imbalance of 5.390977552050073E-5 to a new imbalance of 8.12355507926741E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.006888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:19,295 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1080874717:83 , srv119568927:83 , srv1210816908:83 , srv1269508605:83 , srv1325649140:83 , srv1462158755:83 , srv1635447070:83 , srv2064451594:83 , srv2134688338:83 , srv436528360:83 , srv623802888:83 , srv694268881:83 , srv917469945:83 , srv1023783130:84 , srv102682240:84 , srv106770311:84 , srv1098860636:84 , srv112996079:84 , srv1138406613:84 , srv1210017256:84 , srv1219020934:84 , srv1235655751:84 , srv129902853:84 , srv1338934828:84 , srv1339098630:84 , srv1350141397:84 , srv1374470165:84 , srv1413123205:84 , srv142472650:84 , srv1466750648:84 , srv1481217451:84 , srv1530749803:84 , srv1560628284:84 , srv1576311808:84 , srv1589287317:84 , srv1617160447:84 , srv1623853093:84 , srv1657810459:84 , srv1698131216:84 , srv1748966733:84 , srv1761470354:84 , srv1761733245:84 , srv1764843534:84 , srv1775864274:84 , srv1779837539:84 , srv184125165:84 , srv1852979621:84 , srv1854353846:84 , srv1874874846:84 , srv1908491504:84 , srv1918019972:84 , srv1921934901:84 , srv1992013899:84 , srv1994493517:84 , srv1995793815:84 , srv2019405391:84 , srv2042115383:84 , srv2051714882:84 , srv2138219327:84 , srv214141867:84 , srv236847643:84 , srv260265510:84 , srv334228228:84 , srv373944622:84 , srv376316467:84 , srv379144354:84 , srv39486154:84 , srv402965939:84 , srv412430804:84 , srv415484178:84 , srv424686333:84 , srv490421725:84 , srv511621247:84 , srv557285929:84 , srv587423552:84 , srv590727050:84 , srv60809922:84 , srv639428532:84 , srv649161505:84 , srv661057650:84 , srv717192263:84 , srv724416326:84 , srv756442259:84 , srv789940722:84 , srv806333866:84 , srv858146254:84 , srv1027598662:85 , srv1122339761:85 , srv1268196995:85 , srv164363376:85 , srv1859838797:85 , srv191186670:85 , srv1953872553:85 , srv195658441:85 , srv2019880009:85 , srv2144291918:85 , srv435721843:85 , srv567315261:85 , srv711008607:85 , srv2099251233:684 } 2023-09-23 02:53:19,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:19,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv102682240=1, srv2019880009=58, srv2042115383=59, srv1992013899=54, srv424686333=77, srv184125165=43, srv694268881=91, srv119568927=9, srv1852979621=44, srv1854353846=45, srv236847643=67, srv1859838797=46, srv1918019972=50, srv490421725=80, srv1635447070=33, srv379144354=72, srv590727050=85, srv2019405391=57, srv587423552=84, srv649161505=89, srv1617160447=31, srv1761470354=38, srv1764843534=40, srv1995793815=56, srv639428532=88, srv1122339761=6, srv1874874846=47, srv1466750648=25, srv1481217451=26, srv1775864274=41, srv1589287317=30, srv1219020934=12, srv1235655751=13, srv373944622=70, srv623802888=87, srv557285929=82, srv436528360=79, srv334228228=69, srv1023783130=0, srv1921934901=51, srv2064451594=61, srv1268196995=14, srv1138406613=8, srv1462158755=24, srv789940722=96, srv717192263=93, srv756442259=95, srv1098860636=5, srv1325649140=17, srv1350141397=20, srv60809922=86, srv195658441=53, srv1698131216=36, srv1560628284=28, srv214141867=65, srv858146254=98, srv1413123205=22, srv164363376=34, srv511621247=81, srv2051714882=60, srv412430804=75, srv2138219327=64, srv1338934828=18, srv1576311808=29, srv112996079=7, srv1761733245=39, srv1908491504=48, srv402965939=74, srv2134688338=63, srv1210017256=10, srv260265510=68, srv661057650=90, srv1269508605=15, srv106770311=3, srv1779837539=42, srv711008607=92, srv806333866=97, srv1530749803=27, srv129902853=16, srv1748966733=37, srv1953872553=52, srv1994493517=55, srv2099251233=62, srv1657810459=35, srv1210816908=11, srv1080874717=4, srv724416326=94, srv1623853093=32, srv1027598662=2, srv142472650=23, srv435721843=78, srv567315261=83, srv1374470165=21, srv191186670=49, srv2144291918=66, srv917469945=99, srv415484178=76, srv1339098630=19, srv376316467=71, srv39486154=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-23 02:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-23 02:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-23 02:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-23 02:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-23 02:53:19,430 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:19,430 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1080874717:83 , srv119568927:83 , srv1210816908:83 , srv1269508605:83 , srv1325649140:83 , srv1462158755:83 , srv1635447070:83 , srv2064451594:83 , srv2134688338:83 , srv436528360:83 , srv623802888:83 , srv694268881:83 , srv917469945:83 , srv1023783130:84 , srv102682240:84 , srv106770311:84 , srv1098860636:84 , srv112996079:84 , srv1138406613:84 , srv1210017256:84 , srv1219020934:84 , srv1235655751:84 , srv129902853:84 , srv1338934828:84 , srv1339098630:84 , srv1350141397:84 , srv1374470165:84 , srv1413123205:84 , srv142472650:84 , srv1466750648:84 , srv1481217451:84 , srv1530749803:84 , srv1560628284:84 , srv1576311808:84 , srv1589287317:84 , srv1617160447:84 , srv1623853093:84 , srv1657810459:84 , srv1698131216:84 , srv1748966733:84 , srv1761470354:84 , srv1761733245:84 , srv1764843534:84 , srv1775864274:84 , srv1779837539:84 , srv184125165:84 , srv1852979621:84 , srv1854353846:84 , srv1874874846:84 , srv1908491504:84 , srv1918019972:84 , srv1921934901:84 , srv1992013899:84 , srv1994493517:84 , srv1995793815:84 , srv2019405391:84 , srv2042115383:84 , srv2051714882:84 , srv2138219327:84 , srv214141867:84 , srv236847643:84 , srv260265510:84 , srv334228228:84 , srv373944622:84 , srv376316467:84 , srv379144354:84 , srv39486154:84 , srv402965939:84 , srv412430804:84 , srv415484178:84 , srv424686333:84 , srv490421725:84 , srv511621247:84 , srv557285929:84 , srv587423552:84 , srv590727050:84 , srv60809922:84 , srv639428532:84 , srv649161505:84 , srv661057650:84 , srv717192263:84 , srv724416326:84 , srv756442259:84 , srv789940722:84 , srv806333866:84 , srv858146254:84 , srv1027598662:85 , srv1122339761:85 , srv1268196995:85 , srv164363376:85 , srv1859838797:85 , srv191186670:85 , srv1953872553:85 , srv195658441:85 , srv2019880009:85 , srv2144291918:85 , srv435721843:85 , srv567315261:85 , srv711008607:85 , srv2099251233:684 } 2023-09-23 02:53:19,458 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 362) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=2663 (was 3264) 2023-09-23 02:53:19,467 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=172, AvailableMemoryMB=2663 2023-09-23 02:53:19,468 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-23 02:53:19,468 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-23 02:53:19,478 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1067809817:84 , srv1094250190:84 , srv1189006624:84 , srv1200489608:84 , srv1349386337:84 , srv1356203647:84 , srv1434061441:84 , srv1456982644:84 , srv1532146897:84 , srv1551296039:84 , srv1552456334:84 , srv1659416882:84 , srv1665643722:84 , srv1742343705:84 , srv1783091000:84 , srv187003900:84 , srv2140400343:84 , srv287390896:84 , srv325671012:84 , srv346073366:84 , srv351751092:84 , srv554005326:84 , srv649325306:84 , srv660689232:84 , srv692824389:84 , srv777394242:84 , srv831816256:84 , srv866798476:84 , srv899134758:84 , srv1963923707:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-23 02:53:19,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:53:19,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv866798476=28, srv2140400343=17, srv692824389=25, srv1551296039=9, srv1963923707=16, srv187003900=15, srv1094250190=1, srv1356203647=5, srv1434061441=6, srv1665643722=12, srv1200489608=3, srv1067809817=0, srv351751092=21, srv1552456334=10, srv1783091000=14, srv1456982644=7, srv649325306=23, srv325671012=19, srv831816256=27, srv899134758=29, srv1349386337=4, srv554005326=22, srv1189006624=2, srv777394242=26, srv1659416882=11, srv1742343705=13, srv287390896=18, srv346073366=20, srv660689232=24, srv1532146897=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:53:19,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:53:19,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-23 02:53:19,507 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.143699260547373, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:53:19,507 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912645834942051, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.143699260547373, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-23 02:55:19,492 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5533443 different iterations. Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912645834942051 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:55:19,498 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1067809817:90 , srv1094250190:90 , srv1189006624:90 , srv1200489608:90 , srv1349386337:90 , srv1356203647:90 , srv1434061441:90 , srv1456982644:90 , srv1532146897:90 , srv1551296039:90 , srv1552456334:90 , srv1659416882:90 , srv1665643722:90 , srv1742343705:90 , srv1783091000:90 , srv187003900:90 , srv1963923707:90 , srv2140400343:90 , srv287390896:90 , srv325671012:90 , srv346073366:90 , srv351751092:90 , srv554005326:90 , srv649325306:90 , srv660689232:90 , srv692824389:90 , srv777394242:90 , srv831816256:90 , srv866798476:90 , srv899134758:90 } 2023-09-23 02:55:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-23 02:55:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv866798476=28, srv2140400343=17, srv692824389=25, srv1551296039=9, srv1963923707=16, srv187003900=15, srv1094250190=1, srv1356203647=5, srv1434061441=6, srv1665643722=12, srv1200489608=3, srv1067809817=0, srv351751092=21, srv1552456334=10, srv1783091000=14, srv1456982644=7, srv649325306=23, srv325671012=19, srv831816256=27, srv899134758=29, srv1349386337=4, srv554005326=22, srv1189006624=2, srv777394242=26, srv1659416882=11, srv1742343705=13, srv287390896=18, srv346073366=20, srv660689232=24, srv1532146897=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-23 02:55:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-23 02:55:19,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-23 02:55:19,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-23 02:55:19,517 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-23 02:55:19,517 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1067809817:90 , srv1094250190:90 , srv1189006624:90 , srv1200489608:90 , srv1349386337:90 , srv1356203647:90 , srv1434061441:90 , srv1456982644:90 , srv1532146897:90 , srv1551296039:90 , srv1552456334:90 , srv1659416882:90 , srv1665643722:90 , srv1742343705:90 , srv1783091000:90 , srv187003900:90 , srv1963923707:90 , srv2140400343:90 , srv287390896:90 , srv325671012:90 , srv346073366:90 , srv351751092:90 , srv554005326:90 , srv649325306:90 , srv660689232:90 , srv692824389:90 , srv777394242:90 , srv831816256:90 , srv866798476:90 , srv899134758:90 } 2023-09-23 02:55:19,530 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=159 (was 373), ProcessCount=170 (was 172), AvailableMemoryMB=1718 (was 2663)