2024-04-19 11:53:28,201 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-19 11:53:28,552 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-19 11:53:28,672 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-19 11:53:28,737 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-19 11:53:28,745 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-19 11:53:28,761 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=276, ProcessCount=167, AvailableMemoryMB=2955 2024-04-19 11:53:28,763 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-19 11:53:28,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-19 11:53:28,887 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001926277:84 , srv1038019112:84 , srv1051577682:84 , srv108924843:84 , srv1090116494:84 , srv1103957301:84 , srv1141588573:84 , srv1153137768:84 , srv1155817633:84 , srv1156951958:84 , srv1187385918:84 , srv1194997795:84 , srv1210846521:84 , srv1239285618:84 , srv1243672120:84 , srv128528510:84 , srv1294286551:84 , srv1305568088:84 , srv1333927480:84 , srv1356067875:84 , srv1368594805:84 , srv1396206030:84 , srv1404051506:84 , srv1405100886:84 , srv140807055:84 , srv1411010267:84 , srv1463372917:84 , srv1513008661:84 , srv1522512603:84 , srv1534824454:84 , srv1546934643:84 , srv1565807313:84 , srv1626009756:84 , srv163748372:84 , srv1640525088:84 , srv1648985768:84 , srv1671185121:84 , srv1726331920:84 , srv1752726607:84 , srv1779662189:84 , srv1808417745:84 , srv1808422638:84 , srv1860697174:84 , srv188636556:84 , srv1902847800:84 , srv1916372017:84 , srv1919294224:84 , srv1925157963:84 , srv1926591332:84 , srv1928384211:84 , srv1967926485:84 , srv1971449496:84 , srv2005152174:84 , srv2018154054:84 , srv2028264697:84 , srv2046307513:84 , srv2064198290:84 , srv2088406809:84 , srv2105042248:84 , srv2137043006:84 , srv218093746:84 , srv218736412:84 , srv267991771:84 , srv319335444:84 , srv369435208:84 , srv370612918:84 , srv427469247:84 , srv432194815:84 , srv434435270:84 , srv455495697:84 , srv500702852:84 , srv500736338:84 , srv618684354:84 , srv627053955:84 , srv630796845:84 , srv651598920:84 , srv703252886:84 , srv708468758:84 , srv730232888:84 , srv734247367:84 , srv761924764:84 , srv765747192:84 , srv768871673:84 , srv772683125:84 , srv780801631:84 , srv803315360:84 , srv806456142:84 , srv810602571:84 , srv82685820:84 , srv850832221:84 , srv876640413:84 , srv889586968:84 , srv891563184:84 , srv905997731:84 , srv918262808:84 , srv951592776:84 , srv989230405:84 , srv997490699:84 , srv999019860:84 , srv1998615347:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-19 11:53:28,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:28,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779662189=39, srv2137043006=60, srv1998615347=52, srv218093746=61, srv140807055=24, srv999019860=99, srv1333927480=18, srv1513008661=27, srv500736338=72, srv1411010267=25, srv1648985768=35, srv1928384211=49, srv2105042248=59, srv651598920=76, srv1919294224=46, srv891563184=93, srv708468758=78, srv1902847800=44, srv432194815=68, srv768871673=83, srv1808422638=41, srv997490699=98, srv806456142=87, srv1294286551=16, srv1671185121=36, srv876640413=91, srv427469247=67, srv1141588573=6, srv218736412=62, srv1051577682=2, srv2005152174=53, srv1752726607=38, srv1404051506=22, srv730232888=79, srv1243672120=14, srv2018154054=54, srv1808417745=40, srv765747192=82, srv772683125=84, srv163748372=33, srv1925157963=47, srv951592776=96, srv1368594805=20, srv761924764=81, srv989230405=97, srv1090116494=4, srv434435270=69, srv1396206030=21, srv1001926277=0, srv1239285618=13, srv1971449496=51, srv1860697174=42, srv630796845=75, srv319335444=64, srv1916372017=45, srv2046307513=56, srv803315360=86, srv500702852=71, srv1038019112=1, srv370612918=66, srv1356067875=19, srv369435208=65, srv810602571=88, srv918262808=95, srv2088406809=58, srv618684354=73, srv1926591332=48, srv1187385918=10, srv1153137768=7, srv1463372917=26, srv188636556=43, srv1522512603=28, srv1534824454=29, srv780801631=85, srv734247367=80, srv1194997795=11, srv850832221=90, srv1155817633=8, srv2064198290=57, srv1565807313=31, srv1640525088=34, srv1626009756=32, srv1305568088=17, srv889586968=92, srv2028264697=55, srv1405100886=23, srv905997731=94, srv82685820=89, srv1156951958=9, srv128528510=15, srv627053955=74, srv455495697=70, srv1210846521=12, srv1546934643=30, srv267991771=63, srv108924843=3, srv1967926485=50, srv703252886=77, srv1103957301=5, srv1726331920=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:28,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:28,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:28,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:28,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-19 11:53:28,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-19 11:53:28,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-19 11:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-19 11:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-19 11:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-19 11:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-19 11:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-19 11:53:29,284 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:29,284 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-19 11:53:29,284 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-19 11:53:29,749 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 720 ms to try 5000 different iterations. Found a solution that moves 4640 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18607245619616622. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03880751467064186, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056331948661084916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5155555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03880751467064186, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1815, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24658333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:29,761 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1051577682:83 , srv1141588573:83 , srv1396206030:83 , srv1779662189:83 , srv1916372017:83 , srv218093746:83 , srv500702852:83 , srv905997731:83 , srv108924843:84 , srv1103957301:84 , srv1356067875:84 , srv1513008661:84 , srv1671185121:84 , srv1726331920:84 , srv1928384211:84 , srv2005152174:84 , srv734247367:84 , srv765747192:84 , srv768871673:84 , srv891563184:84 , srv997490699:84 , srv999019860:84 , srv1153137768:85 , srv1155817633:85 , srv163748372:85 , srv188636556:85 , srv267991771:85 , srv432194815:85 , srv627053955:85 , srv708468758:85 , srv730232888:85 , srv772683125:85 , srv810602571:85 , srv82685820:85 , srv876640413:85 , srv951592776:85 , srv1187385918:86 , srv1194997795:86 , srv1210846521:86 , srv1239285618:86 , srv1294286551:86 , srv1333927480:86 , srv1368594805:86 , srv1404051506:86 , srv140807055:86 , srv1463372917:86 , srv1546934643:86 , srv1648985768:86 , srv1919294224:86 , srv1925157963:86 , srv2018154054:86 , srv2046307513:86 , srv2105042248:86 , srv2137043006:86 , srv370612918:86 , srv455495697:86 , srv1038019112:87 , srv1243672120:87 , srv128528510:87 , srv1752726607:87 , srv1902847800:87 , srv2028264697:87 , srv319335444:87 , srv427469247:87 , srv761924764:87 , srv806456142:87 , srv918262808:87 , srv1001926277:88 , srv1156951958:88 , srv1305568088:88 , srv1522512603:88 , srv1808417745:88 , srv1860697174:88 , srv1967926485:88 , srv1971449496:88 , srv2064198290:88 , srv434435270:88 , srv618684354:88 , srv803315360:88 , srv889586968:88 , srv1090116494:89 , srv1565807313:89 , srv1926591332:89 , srv2088406809:89 , srv369435208:89 , srv500736338:89 , srv1405100886:90 , srv1411010267:90 , srv1640525088:90 , srv630796845:90 , srv651598920:90 , srv850832221:90 , srv989230405:90 , srv218736412:91 , srv703252886:91 , srv780801631:91 , srv1626009756:92 , srv1808422638:92 , srv1534824454:93 , srv1998615347:435 } 2024-04-19 11:53:29,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:29,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779662189=39, srv2137043006=60, srv1998615347=52, srv218093746=61, srv140807055=24, srv999019860=99, srv1333927480=18, srv1513008661=27, srv500736338=72, srv1411010267=25, srv1648985768=35, srv1928384211=49, srv2105042248=59, srv651598920=76, srv1919294224=46, srv891563184=93, srv708468758=78, srv1902847800=44, srv432194815=68, srv768871673=83, srv1808422638=41, srv997490699=98, srv806456142=87, srv1294286551=16, srv1671185121=36, srv876640413=91, srv427469247=67, srv1141588573=6, srv218736412=62, srv1051577682=2, srv2005152174=53, srv1752726607=38, srv1404051506=22, srv730232888=79, srv1243672120=14, srv2018154054=54, srv1808417745=40, srv765747192=82, srv772683125=84, srv163748372=33, srv1925157963=47, srv951592776=96, srv1368594805=20, srv761924764=81, srv989230405=97, srv1090116494=4, srv434435270=69, srv1396206030=21, srv1001926277=0, srv1239285618=13, srv1971449496=51, srv1860697174=42, srv630796845=75, srv319335444=64, srv1916372017=45, srv2046307513=56, srv803315360=86, srv500702852=71, srv1038019112=1, srv370612918=66, srv1356067875=19, srv369435208=65, srv810602571=88, srv918262808=95, srv2088406809=58, srv618684354=73, srv1926591332=48, srv1187385918=10, srv1153137768=7, srv1463372917=26, srv188636556=43, srv1522512603=28, srv1534824454=29, srv780801631=85, srv734247367=80, srv1194997795=11, srv850832221=90, srv1155817633=8, srv2064198290=57, srv1565807313=31, srv1640525088=34, srv1626009756=32, srv1305568088=17, srv889586968=92, srv2028264697=55, srv1405100886=23, srv905997731=94, srv82685820=89, srv1156951958=9, srv128528510=15, srv627053955=74, srv455495697=70, srv1210846521=12, srv1546934643=30, srv267991771=63, srv108924843=3, srv1967926485=50, srv703252886=77, srv1103957301=5, srv1726331920=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:29,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:29,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:29,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:29,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:29,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:29,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:29,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:29,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:29,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:29,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:29,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-19 11:53:29,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-19 11:53:29,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-19 11:53:29,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-19 11:53:29,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-19 11:53:29,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-19 11:53:29,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-19 11:53:29,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-19 11:53:29,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-19 11:53:30,137 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03880751467064186, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056331948661084916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03880751467064186, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1815, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24658333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:30,137 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-19 11:53:30,138 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1860399662217441, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03880751467064186, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056331948661084916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03880751467064186, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1815, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24658333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-19 11:53:30,630 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 791 ms to try 5000 different iterations. Found a solution that moves 2637 regions; Going from a computed imbalance of 0.1860399662217441 to a new imbalance of 0.014629829802850583. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021153893680776314); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03500521045070422, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.293, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021153893680776314); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:30,646 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1534824454:79 , srv1808422638:80 , srv780801631:80 , srv128528510:81 , srv1411010267:81 , srv1626009756:81 , srv651598920:81 , srv703252886:81 , srv370612918:82 , srv1001926277:83 , srv1640525088:83 , srv1902847800:83 , srv2088406809:83 , srv2105042248:83 , srv218736412:83 , srv618684354:83 , srv630796845:83 , srv850832221:83 , srv989230405:83 , srv1038019112:84 , srv1090116494:84 , srv1156951958:84 , srv1305568088:84 , srv1405100886:84 , srv1522512603:84 , srv1546934643:84 , srv1565807313:84 , srv1752726607:84 , srv1860697174:84 , srv1971449496:84 , srv2028264697:84 , srv319335444:84 , srv369435208:84 , srv434435270:84 , srv803315360:84 , srv889586968:84 , srv1155817633:85 , srv1243672120:85 , srv1648985768:85 , srv1919294224:85 , srv1926591332:85 , srv1967926485:85 , srv2064198290:85 , srv2137043006:85 , srv500736338:85 , srv772683125:85 , srv810602571:85 , srv108924843:86 , srv1194997795:86 , srv1239285618:86 , srv1294286551:86 , srv1333927480:86 , srv1368594805:86 , srv163748372:86 , srv188636556:86 , srv2018154054:86 , srv627053955:86 , srv761924764:86 , srv806456142:86 , srv1103957301:87 , srv1153137768:87 , srv1210846521:87 , srv140807055:87 , srv1513008661:87 , srv1671185121:87 , srv1808417745:87 , srv1925157963:87 , srv267991771:87 , srv427469247:87 , srv708468758:87 , srv730232888:87 , srv765747192:87 , srv876640413:87 , srv997490699:87 , srv1356067875:88 , srv1404051506:88 , srv1463372917:88 , srv1928384211:88 , srv2046307513:88 , srv432194815:88 , srv734247367:88 , srv768871673:88 , srv82685820:88 , srv891563184:88 , srv905997731:88 , srv918262808:88 , srv951592776:88 , srv1141588573:89 , srv1187385918:89 , srv1396206030:89 , srv2005152174:89 , srv218093746:89 , srv455495697:89 , srv500702852:89 , srv999019860:89 , srv1726331920:90 , srv1779662189:90 , srv1916372017:90 , srv1051577682:91 , srv1998615347:527 } 2024-04-19 11:53:30,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:30,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779662189=39, srv2137043006=60, srv1998615347=52, srv218093746=61, srv140807055=24, srv999019860=99, srv1333927480=18, srv1513008661=27, srv500736338=72, srv1411010267=25, srv1648985768=35, srv1928384211=49, srv2105042248=59, srv651598920=76, srv1919294224=46, srv891563184=93, srv708468758=78, srv1902847800=44, srv432194815=68, srv768871673=83, srv1808422638=41, srv997490699=98, srv806456142=87, srv1294286551=16, srv1671185121=36, srv876640413=91, srv427469247=67, srv1141588573=6, srv218736412=62, srv1051577682=2, srv2005152174=53, srv1752726607=38, srv1404051506=22, srv730232888=79, srv1243672120=14, srv2018154054=54, srv1808417745=40, srv765747192=82, srv772683125=84, srv163748372=33, srv1925157963=47, srv951592776=96, srv1368594805=20, srv761924764=81, srv989230405=97, srv1090116494=4, srv434435270=69, srv1396206030=21, srv1001926277=0, srv1239285618=13, srv1971449496=51, srv1860697174=42, srv630796845=75, srv319335444=64, srv1916372017=45, srv2046307513=56, srv803315360=86, srv500702852=71, srv1038019112=1, srv370612918=66, srv1356067875=19, srv369435208=65, srv810602571=88, srv918262808=95, srv2088406809=58, srv618684354=73, srv1926591332=48, srv1187385918=10, srv1153137768=7, srv1463372917=26, srv188636556=43, srv1522512603=28, srv1534824454=29, srv780801631=85, srv734247367=80, srv1194997795=11, srv850832221=90, srv1155817633=8, srv2064198290=57, srv1565807313=31, srv1640525088=34, srv1626009756=32, srv1305568088=17, srv889586968=92, srv2028264697=55, srv1405100886=23, srv905997731=94, srv82685820=89, srv1156951958=9, srv128528510=15, srv627053955=74, srv455495697=70, srv1210846521=12, srv1546934643=30, srv267991771=63, srv108924843=3, srv1967926485=50, srv703252886=77, srv1103957301=5, srv1726331920=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:30,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:30,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:30,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:30,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:30,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:30,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:30,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:30,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-19 11:53:30,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-19 11:53:30,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-19 11:53:30,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-19 11:53:30,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-19 11:53:30,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-19 11:53:30,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-19 11:53:30,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-19 11:53:30,838 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021153893680776314); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03500521045070422, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021153893680776314); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:30,839 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-19 11:53:30,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014611365134197305, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021153893680776314); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03500521045070422, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021153893680776314); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-19 11:53:31,216 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 512 ms to try 5000 different iterations. Found a solution that moves 996 regions; Going from a computed imbalance of 0.014611365134197305 to a new imbalance of 9.009067487189433E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004441637722065092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005504818825631802); MoveCostFunction : (multiplier=7.0, imbalance=0.11066666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004441637722065092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:31,219 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1187385918:82 , srv1808417745:82 , srv1051577682:83 , srv1404051506:83 , srv1726331920:83 , srv1926591332:83 , srv455495697:83 , srv918262808:83 , srv1210846521:84 , srv1405100886:84 , srv140807055:84 , srv1463372917:84 , srv1640525088:84 , srv1779662189:84 , srv1916372017:84 , srv1967926485:84 , srv2005152174:84 , srv2046307513:84 , srv218736412:84 , srv427469247:84 , srv432194815:84 , srv500702852:84 , srv500736338:84 , srv1090116494:85 , srv1141588573:85 , srv1239285618:85 , srv1294286551:85 , srv1305568088:85 , srv1333927480:85 , srv1356067875:85 , srv1565807313:85 , srv1626009756:85 , srv1925157963:85 , srv1928384211:85 , srv2064198290:85 , srv218093746:85 , srv369435208:85 , srv434435270:85 , srv630796845:85 , srv708468758:85 , srv730232888:85 , srv734247367:85 , srv806456142:85 , srv82685820:85 , srv850832221:85 , srv876640413:85 , srv951592776:85 , srv999019860:85 , srv1001926277:86 , srv1153137768:86 , srv1156951958:86 , srv1194997795:86 , srv1243672120:86 , srv1368594805:86 , srv1396206030:86 , srv1513008661:86 , srv1534824454:86 , srv163748372:86 , srv1648985768:86 , srv1752726607:86 , srv1808422638:86 , srv1860697174:86 , srv188636556:86 , srv1971449496:86 , srv2018154054:86 , srv2088406809:86 , srv2137043006:86 , srv267991771:86 , srv319335444:86 , srv627053955:86 , srv703252886:86 , srv761924764:86 , srv765747192:86 , srv768871673:86 , srv803315360:86 , srv889586968:86 , srv891563184:86 , srv905997731:86 , srv989230405:86 , srv1038019112:87 , srv1411010267:87 , srv1546934643:87 , srv1671185121:87 , srv1902847800:87 , srv1919294224:87 , srv2028264697:87 , srv651598920:87 , srv772683125:87 , srv780801631:87 , srv810602571:87 , srv997490699:87 , srv108924843:88 , srv1103957301:88 , srv1155817633:88 , srv2105042248:88 , srv618684354:88 , srv128528510:89 , srv1522512603:89 , srv370612918:90 , srv1998615347:535 } 2024-04-19 11:53:31,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:31,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779662189=39, srv2137043006=60, srv1998615347=52, srv218093746=61, srv140807055=24, srv999019860=99, srv1333927480=18, srv1513008661=27, srv500736338=72, srv1411010267=25, srv1648985768=35, srv1928384211=49, srv2105042248=59, srv651598920=76, srv1919294224=46, srv891563184=93, srv708468758=78, srv1902847800=44, srv432194815=68, srv768871673=83, srv1808422638=41, srv997490699=98, srv806456142=87, srv1294286551=16, srv1671185121=36, srv876640413=91, srv427469247=67, srv1141588573=6, srv218736412=62, srv1051577682=2, srv2005152174=53, srv1752726607=38, srv1404051506=22, srv730232888=79, srv1243672120=14, srv2018154054=54, srv1808417745=40, srv765747192=82, srv772683125=84, srv163748372=33, srv1925157963=47, srv951592776=96, srv1368594805=20, srv761924764=81, srv989230405=97, srv1090116494=4, srv434435270=69, srv1396206030=21, srv1001926277=0, srv1239285618=13, srv1971449496=51, srv1860697174=42, srv630796845=75, srv319335444=64, srv1916372017=45, srv2046307513=56, srv803315360=86, srv500702852=71, srv1038019112=1, srv370612918=66, srv1356067875=19, srv369435208=65, srv810602571=88, srv918262808=95, srv2088406809=58, srv618684354=73, srv1926591332=48, srv1187385918=10, srv1153137768=7, srv1463372917=26, srv188636556=43, srv1522512603=28, srv1534824454=29, srv780801631=85, srv734247367=80, srv1194997795=11, srv850832221=90, srv1155817633=8, srv2064198290=57, srv1565807313=31, srv1640525088=34, srv1626009756=32, srv1305568088=17, srv889586968=92, srv2028264697=55, srv1405100886=23, srv905997731=94, srv82685820=89, srv1156951958=9, srv128528510=15, srv627053955=74, srv455495697=70, srv1210846521=12, srv1546934643=30, srv267991771=63, srv108924843=3, srv1967926485=50, srv703252886=77, srv1103957301=5, srv1726331920=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:31,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:31,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:31,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:31,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:31,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:31,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:31,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:31,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-19 11:53:31,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-19 11:53:31,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-19 11:53:31,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-19 11:53:31,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-19 11:53:31,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-19 11:53:31,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-19 11:53:31,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-19 11:53:31,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-19 11:53:31,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-19 11:53:31,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-19 11:53:31,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-19 11:53:31,398 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004441637722065092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005504818825631802); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004441637722065092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:31,399 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-19 11:53:31,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.939326076576375E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004441637722065092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005504818825631802); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004441637722065092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-19 11:53:31,881 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 607 ms to try 5000 different iterations. Found a solution that moves 229 regions; Going from a computed imbalance of 8.939326076576375E-4 to a new imbalance of 2.754413087852302E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.025444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:31,883 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1522512603:80 , srv1103957301:83 , srv1671185121:83 , srv1860697174:83 , srv2046307513:83 , srv618684354:83 , srv651598920:83 , srv761924764:83 , srv989230405:83 , srv1038019112:84 , srv108924843:84 , srv1090116494:84 , srv1153137768:84 , srv1155817633:84 , srv1156951958:84 , srv1194997795:84 , srv1243672120:84 , srv1368594805:84 , srv1405100886:84 , srv1411010267:84 , srv1463372917:84 , srv1565807313:84 , srv1626009756:84 , srv1808422638:84 , srv1902847800:84 , srv1919294224:84 , srv1925157963:84 , srv1971449496:84 , srv2018154054:84 , srv2064198290:84 , srv2088406809:84 , srv2137043006:84 , srv218736412:84 , srv370612918:84 , srv427469247:84 , srv455495697:84 , srv500736338:84 , srv627053955:84 , srv703252886:84 , srv730232888:84 , srv768871673:84 , srv772683125:84 , srv780801631:84 , srv803315360:84 , srv806456142:84 , srv82685820:84 , srv850832221:84 , srv889586968:84 , srv891563184:84 , srv951592776:84 , srv997490699:84 , srv999019860:84 , srv1001926277:85 , srv1051577682:85 , srv1141588573:85 , srv1187385918:85 , srv1210846521:85 , srv1239285618:85 , srv128528510:85 , srv1294286551:85 , srv1305568088:85 , srv1333927480:85 , srv1356067875:85 , srv1396206030:85 , srv1404051506:85 , srv140807055:85 , srv1534824454:85 , srv1546934643:85 , srv163748372:85 , srv1640525088:85 , srv1648985768:85 , srv1726331920:85 , srv1752726607:85 , srv1779662189:85 , srv1808417745:85 , srv188636556:85 , srv1916372017:85 , srv1926591332:85 , srv1928384211:85 , srv1967926485:85 , srv2005152174:85 , srv2028264697:85 , srv2105042248:85 , srv218093746:85 , srv267991771:85 , srv319335444:85 , srv369435208:85 , srv432194815:85 , srv630796845:85 , srv708468758:85 , srv734247367:85 , srv765747192:85 , srv810602571:85 , srv876640413:85 , srv905997731:85 , srv918262808:85 , srv1513008661:86 , srv434435270:86 , srv500702852:86 , srv1998615347:646 } 2024-04-19 11:53:31,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:31,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779662189=39, srv2137043006=60, srv1998615347=52, srv218093746=61, srv140807055=24, srv999019860=99, srv1333927480=18, srv1513008661=27, srv500736338=72, srv1411010267=25, srv1648985768=35, srv1928384211=49, srv2105042248=59, srv651598920=76, srv1919294224=46, srv891563184=93, srv708468758=78, srv1902847800=44, srv432194815=68, srv768871673=83, srv1808422638=41, srv997490699=98, srv806456142=87, srv1294286551=16, srv1671185121=36, srv876640413=91, srv427469247=67, srv1141588573=6, srv218736412=62, srv1051577682=2, srv2005152174=53, srv1752726607=38, srv1404051506=22, srv730232888=79, srv1243672120=14, srv2018154054=54, srv1808417745=40, srv765747192=82, srv772683125=84, srv163748372=33, srv1925157963=47, srv951592776=96, srv1368594805=20, srv761924764=81, srv989230405=97, srv1090116494=4, srv434435270=69, srv1396206030=21, srv1001926277=0, srv1239285618=13, srv1971449496=51, srv1860697174=42, srv630796845=75, srv319335444=64, srv1916372017=45, srv2046307513=56, srv803315360=86, srv500702852=71, srv1038019112=1, srv370612918=66, srv1356067875=19, srv369435208=65, srv810602571=88, srv918262808=95, srv2088406809=58, srv618684354=73, srv1926591332=48, srv1187385918=10, srv1153137768=7, srv1463372917=26, srv188636556=43, srv1522512603=28, srv1534824454=29, srv780801631=85, srv734247367=80, srv1194997795=11, srv850832221=90, srv1155817633=8, srv2064198290=57, srv1565807313=31, srv1640525088=34, srv1626009756=32, srv1305568088=17, srv889586968=92, srv2028264697=55, srv1405100886=23, srv905997731=94, srv82685820=89, srv1156951958=9, srv128528510=15, srv627053955=74, srv455495697=70, srv1210846521=12, srv1546934643=30, srv267991771=63, srv108924843=3, srv1967926485=50, srv703252886=77, srv1103957301=5, srv1726331920=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:31,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:31,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:31,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-19 11:53:31,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-19 11:53:31,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-19 11:53:31,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-19 11:53:31,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-19 11:53:31,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-19 11:53:31,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-19 11:53:31,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-19 11:53:31,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-19 11:53:31,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-19 11:53:31,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-19 11:53:31,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-19 11:53:31,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-19 11:53:31,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-19 11:53:31,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-19 11:53:31,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-19 11:53:31,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-19 11:53:31,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-19 11:53:31,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-19 11:53:32,029 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:32,029 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-19 11:53:32,030 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.594063860639548E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-19 11:53:32,275 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 345 ms to try 5000 different iterations. Found a solution that moves 18 regions; Going from a computed imbalance of 2.594063860639548E-5 to a new imbalance of 8.105728810919609E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.002); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:32,276 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1396206030:83 , srv1513008661:83 , srv1534824454:83 , srv2028264697:83 , srv267991771:83 , srv369435208:83 , srv434435270:83 , srv630796845:83 , srv918262808:83 , srv1001926277:84 , srv1038019112:84 , srv1051577682:84 , srv108924843:84 , srv1090116494:84 , srv1103957301:84 , srv1141588573:84 , srv1153137768:84 , srv1155817633:84 , srv1156951958:84 , srv1187385918:84 , srv1194997795:84 , srv1210846521:84 , srv1239285618:84 , srv1243672120:84 , srv128528510:84 , srv1294286551:84 , srv1305568088:84 , srv1333927480:84 , srv1356067875:84 , srv1368594805:84 , srv1404051506:84 , srv1405100886:84 , srv140807055:84 , srv1411010267:84 , srv1463372917:84 , srv1546934643:84 , srv1565807313:84 , srv1626009756:84 , srv163748372:84 , srv1640525088:84 , srv1648985768:84 , srv1726331920:84 , srv1752726607:84 , srv1779662189:84 , srv1808417745:84 , srv1808422638:84 , srv188636556:84 , srv1916372017:84 , srv1919294224:84 , srv1925157963:84 , srv1926591332:84 , srv1928384211:84 , srv1967926485:84 , srv1971449496:84 , srv2005152174:84 , srv2018154054:84 , srv2064198290:84 , srv2088406809:84 , srv2105042248:84 , srv218093746:84 , srv218736412:84 , srv319335444:84 , srv370612918:84 , srv427469247:84 , srv432194815:84 , srv455495697:84 , srv500702852:84 , srv500736338:84 , srv618684354:84 , srv703252886:84 , srv708468758:84 , srv734247367:84 , srv761924764:84 , srv765747192:84 , srv768871673:84 , srv780801631:84 , srv803315360:84 , srv806456142:84 , srv810602571:84 , srv82685820:84 , srv850832221:84 , srv876640413:84 , srv889586968:84 , srv891563184:84 , srv905997731:84 , srv951592776:84 , srv989230405:84 , srv997490699:84 , srv999019860:84 , srv1522512603:85 , srv1671185121:85 , srv1860697174:85 , srv1902847800:85 , srv2046307513:85 , srv2137043006:85 , srv627053955:85 , srv651598920:85 , srv730232888:85 , srv772683125:85 , srv1998615347:683 } 2024-04-19 11:53:32,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:32,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779662189=39, srv2137043006=60, srv1998615347=52, srv218093746=61, srv140807055=24, srv999019860=99, srv1333927480=18, srv1513008661=27, srv500736338=72, srv1411010267=25, srv1648985768=35, srv1928384211=49, srv2105042248=59, srv651598920=76, srv1919294224=46, srv891563184=93, srv708468758=78, srv1902847800=44, srv432194815=68, srv768871673=83, srv1808422638=41, srv997490699=98, srv806456142=87, srv1294286551=16, srv1671185121=36, srv876640413=91, srv427469247=67, srv1141588573=6, srv218736412=62, srv1051577682=2, srv2005152174=53, srv1752726607=38, srv1404051506=22, srv730232888=79, srv1243672120=14, srv2018154054=54, srv1808417745=40, srv765747192=82, srv772683125=84, srv163748372=33, srv1925157963=47, srv951592776=96, srv1368594805=20, srv761924764=81, srv989230405=97, srv1090116494=4, srv434435270=69, srv1396206030=21, srv1001926277=0, srv1239285618=13, srv1971449496=51, srv1860697174=42, srv630796845=75, srv319335444=64, srv1916372017=45, srv2046307513=56, srv803315360=86, srv500702852=71, srv1038019112=1, srv370612918=66, srv1356067875=19, srv369435208=65, srv810602571=88, srv918262808=95, srv2088406809=58, srv618684354=73, srv1926591332=48, srv1187385918=10, srv1153137768=7, srv1463372917=26, srv188636556=43, srv1522512603=28, srv1534824454=29, srv780801631=85, srv734247367=80, srv1194997795=11, srv850832221=90, srv1155817633=8, srv2064198290=57, srv1565807313=31, srv1640525088=34, srv1626009756=32, srv1305568088=17, srv889586968=92, srv2028264697=55, srv1405100886=23, srv905997731=94, srv82685820=89, srv1156951958=9, srv128528510=15, srv627053955=74, srv455495697=70, srv1210846521=12, srv1546934643=30, srv267991771=63, srv108924843=3, srv1967926485=50, srv703252886=77, srv1103957301=5, srv1726331920=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:32,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:32,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:32,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:32,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:32,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-19 11:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-19 11:53:32,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-19 11:53:32,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-19 11:53:32,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-19 11:53:32,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-19 11:53:32,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-19 11:53:32,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-19 11:53:32,412 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:32,412 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1396206030:83 , srv1513008661:83 , srv1534824454:83 , srv2028264697:83 , srv267991771:83 , srv369435208:83 , srv434435270:83 , srv630796845:83 , srv918262808:83 , srv1001926277:84 , srv1038019112:84 , srv1051577682:84 , srv108924843:84 , srv1090116494:84 , srv1103957301:84 , srv1141588573:84 , srv1153137768:84 , srv1155817633:84 , srv1156951958:84 , srv1187385918:84 , srv1194997795:84 , srv1210846521:84 , srv1239285618:84 , srv1243672120:84 , srv128528510:84 , srv1294286551:84 , srv1305568088:84 , srv1333927480:84 , srv1356067875:84 , srv1368594805:84 , srv1404051506:84 , srv1405100886:84 , srv140807055:84 , srv1411010267:84 , srv1463372917:84 , srv1546934643:84 , srv1565807313:84 , srv1626009756:84 , srv163748372:84 , srv1640525088:84 , srv1648985768:84 , srv1726331920:84 , srv1752726607:84 , srv1779662189:84 , srv1808417745:84 , srv1808422638:84 , srv188636556:84 , srv1916372017:84 , srv1919294224:84 , srv1925157963:84 , srv1926591332:84 , srv1928384211:84 , srv1967926485:84 , srv1971449496:84 , srv2005152174:84 , srv2018154054:84 , srv2064198290:84 , srv2088406809:84 , srv2105042248:84 , srv218093746:84 , srv218736412:84 , srv319335444:84 , srv370612918:84 , srv427469247:84 , srv432194815:84 , srv455495697:84 , srv500702852:84 , srv500736338:84 , srv618684354:84 , srv703252886:84 , srv708468758:84 , srv734247367:84 , srv761924764:84 , srv765747192:84 , srv768871673:84 , srv780801631:84 , srv803315360:84 , srv806456142:84 , srv810602571:84 , srv82685820:84 , srv850832221:84 , srv876640413:84 , srv889586968:84 , srv891563184:84 , srv905997731:84 , srv951592776:84 , srv989230405:84 , srv997490699:84 , srv999019860:84 , srv1522512603:85 , srv1671185121:85 , srv1860697174:85 , srv1902847800:85 , srv2046307513:85 , srv2137043006:85 , srv627053955:85 , srv651598920:85 , srv730232888:85 , srv772683125:85 , srv1998615347:683 } 2024-04-19 11:53:32,440 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=318 (was 276) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2157 (was 2955) 2024-04-19 11:53:32,449 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=318, ProcessCount=167, AvailableMemoryMB=2157 2024-04-19 11:53:32,449 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-19 11:53:32,450 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-19 11:53:32,457 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023187183:84 , srv1060842552:84 , srv1097735760:84 , srv1153058563:84 , srv1163965726:84 , srv1259749011:84 , srv134290903:84 , srv1376050619:84 , srv1424750281:84 , srv1476189523:84 , srv1527947797:84 , srv162168598:84 , srv179440977:84 , srv1843937333:84 , srv1849775096:84 , srv1900673348:84 , srv1922334381:84 , srv1950859766:84 , srv1970503618:84 , srv2077175683:84 , srv2127794303:84 , srv254893664:84 , srv259310512:84 , srv301862867:84 , srv341794159:84 , srv468337706:84 , srv478256703:84 , srv554833650:84 , srv898909712:84 , srv2102326475:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-19 11:53:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:53:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1843937333=13, srv1424750281=8, srv2127794303=21, srv1376050619=7, srv1153058563=3, srv134290903=6, srv1097735760=2, srv259310512=23, srv179440977=12, srv1259749011=5, srv1900673348=15, srv468337706=26, srv254893664=22, srv2102326475=20, srv1849775096=14, srv341794159=25, srv1950859766=17, srv1060842552=1, srv1476189523=9, srv162168598=11, srv301862867=24, srv1163965726=4, srv898909712=29, srv2077175683=19, srv554833650=28, srv1023187183=0, srv1922334381=16, srv1970503618=18, srv478256703=27, srv1527947797=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:53:32,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:53:32,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:53:32,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:53:32,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:53:32,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:53:32,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:53:32,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:53:32,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:53:32,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-19 11:53:32,485 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1373344947828596, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:53:32,485 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912625779772957, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1373344947828596, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-19 11:55:32,473 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5448517 different iterations. Found a solution that moves 1873 regions; Going from a computed imbalance of 0.9912625779772957 to a new imbalance of 4.3716754376927055E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6937037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:55:32,481 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1023187183:90 , srv1060842552:90 , srv1097735760:90 , srv1153058563:90 , srv1163965726:90 , srv1259749011:90 , srv134290903:90 , srv1376050619:90 , srv1424750281:90 , srv1476189523:90 , srv1527947797:90 , srv162168598:90 , srv179440977:90 , srv1843937333:90 , srv1849775096:90 , srv1900673348:90 , srv1922334381:90 , srv1950859766:90 , srv1970503618:90 , srv2077175683:90 , srv2102326475:90 , srv2127794303:90 , srv254893664:90 , srv259310512:90 , srv301862867:90 , srv341794159:90 , srv468337706:90 , srv478256703:90 , srv554833650:90 , srv898909712:90 } 2024-04-19 11:55:32,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-19 11:55:32,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1843937333=13, srv1424750281=8, srv2127794303=21, srv1376050619=7, srv1153058563=3, srv134290903=6, srv1097735760=2, srv259310512=23, srv179440977=12, srv1259749011=5, srv1900673348=15, srv468337706=26, srv254893664=22, srv2102326475=20, srv1849775096=14, srv341794159=25, srv1950859766=17, srv1060842552=1, srv1476189523=9, srv162168598=11, srv301862867=24, srv1163965726=4, srv898909712=29, srv2077175683=19, srv554833650=28, srv1023187183=0, srv1922334381=16, srv1970503618=18, srv478256703=27, srv1527947797=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-19 11:55:32,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-19 11:55:32,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-19 11:55:32,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-19 11:55:32,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-19 11:55:32,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-19 11:55:32,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-19 11:55:32,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-19 11:55:32,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-19 11:55:32,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-19 11:55:32,507 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-19 11:55:32,508 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1023187183:90 , srv1060842552:90 , srv1097735760:90 , srv1153058563:90 , srv1163965726:90 , srv1259749011:90 , srv134290903:90 , srv1376050619:90 , srv1424750281:90 , srv1476189523:90 , srv1527947797:90 , srv162168598:90 , srv179440977:90 , srv1843937333:90 , srv1849775096:90 , srv1900673348:90 , srv1922334381:90 , srv1950859766:90 , srv1970503618:90 , srv2077175683:90 , srv2102326475:90 , srv2127794303:90 , srv254893664:90 , srv259310512:90 , srv301862867:90 , srv341794159:90 , srv468337706:90 , srv478256703:90 , srv554833650:90 , srv898909712:90 } 2024-04-19 11:55:32,524 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=245 (was 318), ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=1452 (was 2157)