2024-04-23 10:53:29,229 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-23 10:53:29,515 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-23 10:53:29,625 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-23 10:53:29,691 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-23 10:53:29,699 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-23 10:53:29,716 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=276, ProcessCount=170, AvailableMemoryMB=14157 2024-04-23 10:53:29,718 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-23 10:53:29,719 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-23 10:53:29,849 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007707984:84 , srv1022566736:84 , srv1042485528:84 , srv1049143847:84 , srv1072745872:84 , srv1108834109:84 , srv1130555411:84 , srv1135506751:84 , srv1140857428:84 , srv1185640822:84 , srv1188277185:84 , srv1189025658:84 , srv1210499962:84 , srv1253093159:84 , srv1265268468:84 , srv1296857680:84 , srv1317698167:84 , srv1331244514:84 , srv1331299282:84 , srv1338164955:84 , srv1384518287:84 , srv1385475148:84 , srv1394297707:84 , srv1423482415:84 , srv1448772849:84 , srv1456637231:84 , srv1507883942:84 , srv1533019783:84 , srv1559351532:84 , srv1580184826:84 , srv1592050847:84 , srv1601126092:84 , srv1629225167:84 , srv1632561305:84 , srv168667628:84 , srv1703827770:84 , srv1706435144:84 , srv1714273460:84 , srv1723783444:84 , srv1742391300:84 , srv1756955675:84 , srv1768003453:84 , srv177462188:84 , srv1784723395:84 , srv1817610745:84 , srv1818549909:84 , srv1857895624:84 , srv1879436180:84 , srv1895069021:84 , srv1904143656:84 , srv1911470028:84 , srv1957970080:84 , srv1975689434:84 , srv1978321636:84 , srv198824288:84 , srv2001410175:84 , srv2007940100:84 , srv2073213128:84 , srv2074862807:84 , srv2087364623:84 , srv209640103:84 , srv2114705016:84 , srv2134493670:84 , srv247395810:84 , srv25371597:84 , srv255927781:84 , srv289826033:84 , srv358169688:84 , srv387609646:84 , srv410247104:84 , srv449625734:84 , srv451359680:84 , srv473132547:84 , srv517695178:84 , srv518132661:84 , srv536779855:84 , srv541188727:84 , srv592990251:84 , srv593951847:84 , srv59774876:84 , srv628920660:84 , srv632907398:84 , srv653202709:84 , srv735188718:84 , srv756377171:84 , srv817333939:84 , srv821590934:84 , srv822787027:84 , srv82400727:84 , srv824529466:84 , srv834937921:84 , srv88207474:84 , srv909343654:84 , srv928189044:84 , srv931188025:84 , srv93775933:84 , srv941656270:84 , srv994194064:84 , srv999967856:84 , srv1417509809:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-23 10:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1756955675=41, srv1879436180=48, srv88207474=92, srv2087364623=60, srv473132547=73, srv941656270=97, srv59774876=80, srv2074862807=59, srv2001410175=56, srv756377171=85, srv1580184826=30, srv1210499962=12, srv632907398=82, srv1185640822=9, srv517695178=74, srv1022566736=1, srv1895069021=49, srv994194064=98, srv168667628=35, srv1601126092=32, srv536779855=76, srv1140857428=8, srv1975689434=53, srv834937921=91, srv289826033=67, srv1384518287=20, srv1331299282=18, srv541188727=77, srv449625734=71, srv1629225167=33, srv387609646=69, srv1189025658=11, srv1135506751=7, srv593951847=79, srv1007707984=0, srv999967856=99, srv1042485528=2, srv1296857680=15, srv1423482415=24, srv1072745872=4, srv1559351532=29, srv1385475148=21, srv1448772849=25, srv1592050847=31, srv1456637231=26, srv25371597=65, srv1394297707=22, srv1706435144=37, srv255927781=66, srv410247104=70, srv198824288=55, srv1338164955=19, srv1904143656=50, srv1632561305=34, srv1703827770=36, srv1742391300=40, srv1911470028=51, srv1507883942=27, srv518132661=75, srv93775933=96, srv1265268468=14, srv817333939=86, srv928189044=94, srv735188718=84, srv1714273460=38, srv1417509809=23, srv247395810=64, srv1130555411=6, srv209640103=61, srv931188025=95, srv824529466=90, srv1331244514=17, srv1049143847=3, srv1957970080=52, srv1108834109=5, srv1317698167=16, srv909343654=93, srv821590934=87, srv177462188=43, srv1818549909=46, srv358169688=68, srv451359680=72, srv822787027=88, srv82400727=89, srv1533019783=28, srv2073213128=58, srv628920660=81, srv1723783444=39, srv1188277185=10, srv1784723395=44, srv1857895624=47, srv2007940100=57, srv1978321636=54, srv1253093159=13, srv1817610745=45, srv653202709=83, srv2134493670=63, srv592990251=78, srv2114705016=62, srv1768003453=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:29,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:29,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-23 10:53:29,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-23 10:53:29,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-23 10:53:29,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-23 10:53:29,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-23 10:53:29,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-23 10:53:29,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-23 10:53:29,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-23 10:53:29,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-23 10:53:29,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-23 10:53:29,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-23 10:53:29,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-23 10:53:29,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-23 10:53:29,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-23 10:53:29,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-23 10:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-23 10:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-23 10:53:30,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:30,188 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-23 10:53:30,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-23 10:53:30,938 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 916 ms to try 5000 different iterations. Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1712803229556736. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03902860903532591, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053206920893587056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03902860903532591, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16666666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23075, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:30,951 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv994194064:80 , srv1703827770:81 , srv1706435144:81 , srv517695178:81 , srv824529466:81 , srv1857895624:82 , srv255927781:82 , srv1185640822:83 , srv1331244514:83 , srv1331299282:83 , srv1385475148:83 , srv1879436180:83 , srv1975689434:83 , srv2073213128:83 , srv928189044:83 , srv1007707984:84 , srv1049143847:84 , srv1072745872:84 , srv1580184826:84 , srv1818549909:84 , srv1895069021:84 , srv1904143656:84 , srv2114705016:84 , srv25371597:84 , srv289826033:84 , srv358169688:84 , srv817333939:84 , srv821590934:84 , srv88207474:84 , srv1130555411:85 , srv1394297707:85 , srv1601126092:85 , srv168667628:85 , srv1723783444:85 , srv1911470028:85 , srv247395810:85 , srv410247104:85 , srv541188727:85 , srv59774876:85 , srv628920660:85 , srv735188718:85 , srv909343654:85 , srv1022566736:86 , srv1108834109:86 , srv1338164955:86 , srv1384518287:86 , srv1559351532:86 , srv1632561305:86 , srv1714273460:86 , srv1742391300:86 , srv1756955675:86 , srv198824288:86 , srv2001410175:86 , srv387609646:86 , srv451359680:86 , srv536779855:86 , srv593951847:86 , srv632907398:86 , srv834937921:86 , srv1188277185:87 , srv1253093159:87 , srv1423482415:87 , srv1629225167:87 , srv1768003453:87 , srv177462188:87 , srv1817610745:87 , srv592990251:87 , srv653202709:87 , srv999967856:87 , srv1042485528:88 , srv1140857428:88 , srv1189025658:88 , srv1265268468:88 , srv1448772849:88 , srv1592050847:88 , srv2007940100:88 , srv209640103:88 , srv931188025:88 , srv1317698167:89 , srv2087364623:89 , srv473132547:89 , srv756377171:89 , srv1135506751:90 , srv1784723395:90 , srv2074862807:90 , srv822787027:90 , srv82400727:90 , srv1296857680:91 , srv1456637231:91 , srv2134493670:91 , srv1210499962:92 , srv941656270:92 , srv1533019783:93 , srv1957970080:93 , srv449625734:93 , srv518132661:93 , srv1507883942:94 , srv93775933:95 , srv1978321636:103 , srv1417509809:436 } 2024-04-23 10:53:30,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:30,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1756955675=41, srv1879436180=48, srv88207474=92, srv2087364623=60, srv473132547=73, srv941656270=97, srv59774876=80, srv2074862807=59, srv2001410175=56, srv756377171=85, srv1580184826=30, srv1210499962=12, srv632907398=82, srv1185640822=9, srv517695178=74, srv1022566736=1, srv1895069021=49, srv994194064=98, srv168667628=35, srv1601126092=32, srv536779855=76, srv1140857428=8, srv1975689434=53, srv834937921=91, srv289826033=67, srv1384518287=20, srv1331299282=18, srv541188727=77, srv449625734=71, srv1629225167=33, srv387609646=69, srv1189025658=11, srv1135506751=7, srv593951847=79, srv1007707984=0, srv999967856=99, srv1042485528=2, srv1296857680=15, srv1423482415=24, srv1072745872=4, srv1559351532=29, srv1385475148=21, srv1448772849=25, srv1592050847=31, srv1456637231=26, srv25371597=65, srv1394297707=22, srv1706435144=37, srv255927781=66, srv410247104=70, srv198824288=55, srv1338164955=19, srv1904143656=50, srv1632561305=34, srv1703827770=36, srv1742391300=40, srv1911470028=51, srv1507883942=27, srv518132661=75, srv93775933=96, srv1265268468=14, srv817333939=86, srv928189044=94, srv735188718=84, srv1714273460=38, srv1417509809=23, srv247395810=64, srv1130555411=6, srv209640103=61, srv931188025=95, srv824529466=90, srv1331244514=17, srv1049143847=3, srv1957970080=52, srv1108834109=5, srv1317698167=16, srv909343654=93, srv821590934=87, srv177462188=43, srv1818549909=46, srv358169688=68, srv451359680=72, srv822787027=88, srv82400727=89, srv1533019783=28, srv2073213128=58, srv628920660=81, srv1723783444=39, srv1188277185=10, srv1784723395=44, srv1857895624=47, srv2007940100=57, srv1978321636=54, srv1253093159=13, srv1817610745=45, srv653202709=83, srv2134493670=63, srv592990251=78, srv2114705016=62, srv1768003453=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:30,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:30,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:30,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:30,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:30,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-23 10:53:30,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-23 10:53:30,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-23 10:53:30,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-23 10:53:30,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-23 10:53:30,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-23 10:53:30,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-23 10:53:30,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-23 10:53:30,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-23 10:53:31,120 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03902860903532591, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053206920893587056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03902860903532591, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16666666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23075, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:31,120 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-23 10:53:31,120 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17124742685657118, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03902860903532591, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053206920893587056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03902860903532591, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16666666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23075, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-23 10:53:31,388 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 373 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.17124742685657118 to a new imbalance of 0.01387340190087876. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020023679709181754); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029878166376773513, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020023679709181754); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:31,397 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1978321636:68 , srv93775933:75 , srv518132661:78 , srv1296857680:79 , srv1507883942:79 , srv1533019783:79 , srv449625734:79 , srv1957970080:80 , srv1456637231:81 , srv2074862807:81 , srv941656270:81 , srv2134493670:82 , srv1135506751:83 , srv1210499962:83 , srv1317698167:83 , srv1448772849:83 , srv1784723395:83 , srv2087364623:83 , srv473132547:83 , srv756377171:83 , srv82400727:83 , srv1140857428:84 , srv1592050847:84 , srv1632561305:84 , srv1768003453:84 , srv177462188:84 , srv2007940100:84 , srv822787027:84 , srv1042485528:85 , srv1108834109:85 , srv1188277185:85 , srv1338164955:85 , srv1601126092:85 , srv1817610745:85 , srv209640103:85 , srv632907398:85 , srv834937921:85 , srv931188025:85 , srv999967856:85 , srv1022566736:86 , srv1189025658:86 , srv1253093159:86 , srv1265268468:86 , srv1629225167:86 , srv1714273460:86 , srv1723783444:86 , srv1742391300:86 , srv1756955675:86 , srv198824288:86 , srv247395810:86 , srv387609646:86 , srv536779855:86 , srv541188727:86 , srv593951847:86 , srv59774876:86 , srv735188718:86 , srv909343654:86 , srv1007707984:87 , srv1130555411:87 , srv1384518287:87 , srv1423482415:87 , srv1580184826:87 , srv168667628:87 , srv1879436180:87 , srv1904143656:87 , srv2001410175:87 , srv2114705016:87 , srv451359680:87 , srv628920660:87 , srv653202709:87 , srv821590934:87 , srv1072745872:88 , srv1385475148:88 , srv1394297707:88 , srv1857895624:88 , srv1895069021:88 , srv1911470028:88 , srv1975689434:88 , srv410247104:88 , srv592990251:88 , srv88207474:88 , srv1331244514:89 , srv1559351532:89 , srv1818549909:89 , srv2073213128:89 , srv25371597:89 , srv358169688:89 , srv1049143847:90 , srv1185640822:90 , srv1703827770:90 , srv289826033:90 , srv817333939:90 , srv1331299282:91 , srv517695178:91 , srv928189044:91 , srv994194064:91 , srv1706435144:92 , srv255927781:92 , srv824529466:94 , srv1417509809:516 } 2024-04-23 10:53:31,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:31,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1756955675=41, srv1879436180=48, srv88207474=92, srv2087364623=60, srv473132547=73, srv941656270=97, srv59774876=80, srv2074862807=59, srv2001410175=56, srv756377171=85, srv1580184826=30, srv1210499962=12, srv632907398=82, srv1185640822=9, srv517695178=74, srv1022566736=1, srv1895069021=49, srv994194064=98, srv168667628=35, srv1601126092=32, srv536779855=76, srv1140857428=8, srv1975689434=53, srv834937921=91, srv289826033=67, srv1384518287=20, srv1331299282=18, srv541188727=77, srv449625734=71, srv1629225167=33, srv387609646=69, srv1189025658=11, srv1135506751=7, srv593951847=79, srv1007707984=0, srv999967856=99, srv1042485528=2, srv1296857680=15, srv1423482415=24, srv1072745872=4, srv1559351532=29, srv1385475148=21, srv1448772849=25, srv1592050847=31, srv1456637231=26, srv25371597=65, srv1394297707=22, srv1706435144=37, srv255927781=66, srv410247104=70, srv198824288=55, srv1338164955=19, srv1904143656=50, srv1632561305=34, srv1703827770=36, srv1742391300=40, srv1911470028=51, srv1507883942=27, srv518132661=75, srv93775933=96, srv1265268468=14, srv817333939=86, srv928189044=94, srv735188718=84, srv1714273460=38, srv1417509809=23, srv247395810=64, srv1130555411=6, srv209640103=61, srv931188025=95, srv824529466=90, srv1331244514=17, srv1049143847=3, srv1957970080=52, srv1108834109=5, srv1317698167=16, srv909343654=93, srv821590934=87, srv177462188=43, srv1818549909=46, srv358169688=68, srv451359680=72, srv822787027=88, srv82400727=89, srv1533019783=28, srv2073213128=58, srv628920660=81, srv1723783444=39, srv1188277185=10, srv1784723395=44, srv1857895624=47, srv2007940100=57, srv1978321636=54, srv1253093159=13, srv1817610745=45, srv653202709=83, srv2134493670=63, srv592990251=78, srv2114705016=62, srv1768003453=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:31,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:31,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:31,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:31,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:31,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:31,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:31,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-23 10:53:31,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-23 10:53:31,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-23 10:53:31,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-23 10:53:31,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-23 10:53:31,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-23 10:53:31,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-23 10:53:31,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-23 10:53:31,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-23 10:53:31,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-23 10:53:31,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-23 10:53:31,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-23 10:53:31,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-23 10:53:31,549 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020023679709181754); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029878166376773513, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020023679709181754); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:31,549 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-23 10:53:31,550 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01385511928811664, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020023679709181754); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029878166376773513, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020023679709181754); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-23 10:53:31,791 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 348 ms to try 5000 different iterations. Found a solution that moves 1003 regions; Going from a computed imbalance of 0.01385511928811664 to a new imbalance of 4.644138405511265E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004994073055283274); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.11144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004994073055283274); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:31,793 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1559351532:81 , srv824529466:82 , srv1049143847:83 , srv1185640822:83 , srv1331299282:83 , srv1423482415:83 , srv1507883942:83 , srv255927781:83 , srv592990251:83 , srv1189025658:84 , srv1210499962:84 , srv1265268468:84 , srv1394297707:84 , srv1706435144:84 , srv1911470028:84 , srv1957970080:84 , srv209640103:84 , srv25371597:84 , srv410247104:84 , srv451359680:84 , srv653202709:84 , srv817333939:84 , srv822787027:84 , srv82400727:84 , srv928189044:84 , srv931188025:84 , srv1022566736:85 , srv1042485528:85 , srv1072745872:85 , srv1130555411:85 , srv1135506751:85 , srv1253093159:85 , srv1317698167:85 , srv1331244514:85 , srv1592050847:85 , srv1629225167:85 , srv1714273460:85 , srv1742391300:85 , srv1756955675:85 , srv1784723395:85 , srv1817610745:85 , srv1818549909:85 , srv2001410175:85 , srv2007940100:85 , srv2073213128:85 , srv2134493670:85 , srv289826033:85 , srv358169688:85 , srv517695178:85 , srv628920660:85 , srv909343654:85 , srv941656270:85 , srv1108834109:86 , srv1140857428:86 , srv1384518287:86 , srv1385475148:86 , srv1448772849:86 , srv1456637231:86 , srv1533019783:86 , srv1580184826:86 , srv168667628:86 , srv1703827770:86 , srv1768003453:86 , srv177462188:86 , srv1895069021:86 , srv1975689434:86 , srv1978321636:86 , srv198824288:86 , srv2087364623:86 , srv2114705016:86 , srv387609646:86 , srv449625734:86 , srv473132547:86 , srv536779855:86 , srv59774876:86 , srv735188718:86 , srv756377171:86 , srv88207474:86 , srv999967856:86 , srv1007707984:87 , srv1188277185:87 , srv1338164955:87 , srv1723783444:87 , srv1857895624:87 , srv1904143656:87 , srv2074862807:87 , srv247395810:87 , srv518132661:87 , srv541188727:87 , srv593951847:87 , srv632907398:87 , srv821590934:87 , srv834937921:87 , srv1296857680:88 , srv1601126092:88 , srv1632561305:88 , srv1879436180:88 , srv93775933:88 , srv994194064:88 , srv1417509809:550 } 2024-04-23 10:53:31,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:31,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1756955675=41, srv1879436180=48, srv88207474=92, srv2087364623=60, srv473132547=73, srv941656270=97, srv59774876=80, srv2074862807=59, srv2001410175=56, srv756377171=85, srv1580184826=30, srv1210499962=12, srv632907398=82, srv1185640822=9, srv517695178=74, srv1022566736=1, srv1895069021=49, srv994194064=98, srv168667628=35, srv1601126092=32, srv536779855=76, srv1140857428=8, srv1975689434=53, srv834937921=91, srv289826033=67, srv1384518287=20, srv1331299282=18, srv541188727=77, srv449625734=71, srv1629225167=33, srv387609646=69, srv1189025658=11, srv1135506751=7, srv593951847=79, srv1007707984=0, srv999967856=99, srv1042485528=2, srv1296857680=15, srv1423482415=24, srv1072745872=4, srv1559351532=29, srv1385475148=21, srv1448772849=25, srv1592050847=31, srv1456637231=26, srv25371597=65, srv1394297707=22, srv1706435144=37, srv255927781=66, srv410247104=70, srv198824288=55, srv1338164955=19, srv1904143656=50, srv1632561305=34, srv1703827770=36, srv1742391300=40, srv1911470028=51, srv1507883942=27, srv518132661=75, srv93775933=96, srv1265268468=14, srv817333939=86, srv928189044=94, srv735188718=84, srv1714273460=38, srv1417509809=23, srv247395810=64, srv1130555411=6, srv209640103=61, srv931188025=95, srv824529466=90, srv1331244514=17, srv1049143847=3, srv1957970080=52, srv1108834109=5, srv1317698167=16, srv909343654=93, srv821590934=87, srv177462188=43, srv1818549909=46, srv358169688=68, srv451359680=72, srv822787027=88, srv82400727=89, srv1533019783=28, srv2073213128=58, srv628920660=81, srv1723783444=39, srv1188277185=10, srv1784723395=44, srv1857895624=47, srv2007940100=57, srv1978321636=54, srv1253093159=13, srv1817610745=45, srv653202709=83, srv2134493670=63, srv592990251=78, srv2114705016=62, srv1768003453=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:31,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:31,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:31,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-23 10:53:31,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-23 10:53:31,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-23 10:53:31,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-23 10:53:31,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-23 10:53:31,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-23 10:53:31,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-23 10:53:31,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-23 10:53:31,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004994073055283274); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004994073055283274); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:31,923 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-23 10:53:31,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.5739068444220106E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004994073055283274); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004994073055283274); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-23 10:53:32,289 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations. Found a solution that moves 235 regions; Going from a computed imbalance of 4.5739068444220106E-4 to a new imbalance of 2.7035186330497016E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.026111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:32,290 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1188277185:83 , srv1384518287:83 , srv1723783444:83 , srv1904143656:83 , srv2074862807:83 , srv2134493670:83 , srv289826033:83 , srv593951847:83 , srv834937921:83 , srv994194064:83 , srv1042485528:84 , srv1049143847:84 , srv1135506751:84 , srv1140857428:84 , srv1189025658:84 , srv1210499962:84 , srv1253093159:84 , srv1265268468:84 , srv1296857680:84 , srv1456637231:84 , srv1533019783:84 , srv1601126092:84 , srv1629225167:84 , srv1632561305:84 , srv168667628:84 , srv1756955675:84 , srv1784723395:84 , srv1818549909:84 , srv1879436180:84 , srv1895069021:84 , srv1975689434:84 , srv198824288:84 , srv2087364623:84 , srv247395810:84 , srv358169688:84 , srv387609646:84 , srv449625734:84 , srv473132547:84 , srv518132661:84 , srv536779855:84 , srv541188727:84 , srv592990251:84 , srv632907398:84 , srv653202709:84 , srv756377171:84 , srv821590934:84 , srv822787027:84 , srv82400727:84 , srv928189044:84 , srv931188025:84 , srv93775933:84 , srv941656270:84 , srv999967856:84 , srv1007707984:85 , srv1022566736:85 , srv1072745872:85 , srv1108834109:85 , srv1130555411:85 , srv1317698167:85 , srv1331244514:85 , srv1331299282:85 , srv1338164955:85 , srv1385475148:85 , srv1394297707:85 , srv1423482415:85 , srv1448772849:85 , srv1580184826:85 , srv1592050847:85 , srv1703827770:85 , srv1714273460:85 , srv1742391300:85 , srv177462188:85 , srv1817610745:85 , srv1857895624:85 , srv1911470028:85 , srv1957970080:85 , srv1978321636:85 , srv2001410175:85 , srv2007940100:85 , srv2073213128:85 , srv2114705016:85 , srv25371597:85 , srv255927781:85 , srv410247104:85 , srv451359680:85 , srv59774876:85 , srv628920660:85 , srv735188718:85 , srv817333939:85 , srv824529466:85 , srv88207474:85 , srv1185640822:86 , srv1507883942:86 , srv1559351532:86 , srv1706435144:86 , srv1768003453:86 , srv209640103:86 , srv517695178:86 , srv909343654:86 , srv1417509809:640 } 2024-04-23 10:53:32,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:32,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1756955675=41, srv1879436180=48, srv88207474=92, srv2087364623=60, srv473132547=73, srv941656270=97, srv59774876=80, srv2074862807=59, srv2001410175=56, srv756377171=85, srv1580184826=30, srv1210499962=12, srv632907398=82, srv1185640822=9, srv517695178=74, srv1022566736=1, srv1895069021=49, srv994194064=98, srv168667628=35, srv1601126092=32, srv536779855=76, srv1140857428=8, srv1975689434=53, srv834937921=91, srv289826033=67, srv1384518287=20, srv1331299282=18, srv541188727=77, srv449625734=71, srv1629225167=33, srv387609646=69, srv1189025658=11, srv1135506751=7, srv593951847=79, srv1007707984=0, srv999967856=99, srv1042485528=2, srv1296857680=15, srv1423482415=24, srv1072745872=4, srv1559351532=29, srv1385475148=21, srv1448772849=25, srv1592050847=31, srv1456637231=26, srv25371597=65, srv1394297707=22, srv1706435144=37, srv255927781=66, srv410247104=70, srv198824288=55, srv1338164955=19, srv1904143656=50, srv1632561305=34, srv1703827770=36, srv1742391300=40, srv1911470028=51, srv1507883942=27, srv518132661=75, srv93775933=96, srv1265268468=14, srv817333939=86, srv928189044=94, srv735188718=84, srv1714273460=38, srv1417509809=23, srv247395810=64, srv1130555411=6, srv209640103=61, srv931188025=95, srv824529466=90, srv1331244514=17, srv1049143847=3, srv1957970080=52, srv1108834109=5, srv1317698167=16, srv909343654=93, srv821590934=87, srv177462188=43, srv1818549909=46, srv358169688=68, srv451359680=72, srv822787027=88, srv82400727=89, srv1533019783=28, srv2073213128=58, srv628920660=81, srv1723783444=39, srv1188277185=10, srv1784723395=44, srv1857895624=47, srv2007940100=57, srv1978321636=54, srv1253093159=13, srv1817610745=45, srv653202709=83, srv2134493670=63, srv592990251=78, srv2114705016=62, srv1768003453=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:32,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:32,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:32,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:32,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:32,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:32,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:32,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-23 10:53:32,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-23 10:53:32,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-23 10:53:32,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-23 10:53:32,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-23 10:53:32,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-23 10:53:32,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-23 10:53:32,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-23 10:53:32,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:32,472 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-23 10:53:32,473 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.53896811604098E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-23 10:53:32,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations. Found a solution that moves 25 regions; Going from a computed imbalance of 2.53896811604098E-5 to a new imbalance of 6.20713248122701E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.002777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:32,764 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007707984:83 , srv1210499962:83 , srv1338164955:83 , srv1706435144:83 , srv1768003453:83 , srv2001410175:83 , srv209640103:83 , srv517695178:83 , srv817333939:83 , srv88207474:83 , srv1022566736:84 , srv1042485528:84 , srv1072745872:84 , srv1108834109:84 , srv1130555411:84 , srv1135506751:84 , srv1140857428:84 , srv1185640822:84 , srv1188277185:84 , srv1189025658:84 , srv1253093159:84 , srv1265268468:84 , srv1296857680:84 , srv1317698167:84 , srv1331244514:84 , srv1331299282:84 , srv1384518287:84 , srv1385475148:84 , srv1394297707:84 , srv1423482415:84 , srv1448772849:84 , srv1456637231:84 , srv1507883942:84 , srv1533019783:84 , srv1559351532:84 , srv1580184826:84 , srv1592050847:84 , srv1601126092:84 , srv1629225167:84 , srv1632561305:84 , srv168667628:84 , srv1703827770:84 , srv1714273460:84 , srv1742391300:84 , srv177462188:84 , srv1784723395:84 , srv1817610745:84 , srv1818549909:84 , srv1857895624:84 , srv1879436180:84 , srv1895069021:84 , srv1911470028:84 , srv1957970080:84 , srv1978321636:84 , srv198824288:84 , srv2007940100:84 , srv2073213128:84 , srv2087364623:84 , srv2114705016:84 , srv247395810:84 , srv25371597:84 , srv255927781:84 , srv289826033:84 , srv358169688:84 , srv387609646:84 , srv410247104:84 , srv449625734:84 , srv451359680:84 , srv473132547:84 , srv518132661:84 , srv536779855:84 , srv541188727:84 , srv592990251:84 , srv593951847:84 , srv59774876:84 , srv628920660:84 , srv632907398:84 , srv653202709:84 , srv735188718:84 , srv756377171:84 , srv821590934:84 , srv822787027:84 , srv824529466:84 , srv909343654:84 , srv928189044:84 , srv93775933:84 , srv941656270:84 , srv994194064:84 , srv999967856:84 , srv1049143847:85 , srv1723783444:85 , srv1756955675:85 , srv1904143656:85 , srv1975689434:85 , srv2074862807:85 , srv2134493670:85 , srv82400727:85 , srv834937921:85 , srv931188025:85 , srv1417509809:684 } 2024-04-23 10:53:32,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:32,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1756955675=41, srv1879436180=48, srv88207474=92, srv2087364623=60, srv473132547=73, srv941656270=97, srv59774876=80, srv2074862807=59, srv2001410175=56, srv756377171=85, srv1580184826=30, srv1210499962=12, srv632907398=82, srv1185640822=9, srv517695178=74, srv1022566736=1, srv1895069021=49, srv994194064=98, srv168667628=35, srv1601126092=32, srv536779855=76, srv1140857428=8, srv1975689434=53, srv834937921=91, srv289826033=67, srv1384518287=20, srv1331299282=18, srv541188727=77, srv449625734=71, srv1629225167=33, srv387609646=69, srv1189025658=11, srv1135506751=7, srv593951847=79, srv1007707984=0, srv999967856=99, srv1042485528=2, srv1296857680=15, srv1423482415=24, srv1072745872=4, srv1559351532=29, srv1385475148=21, srv1448772849=25, srv1592050847=31, srv1456637231=26, srv25371597=65, srv1394297707=22, srv1706435144=37, srv255927781=66, srv410247104=70, srv198824288=55, srv1338164955=19, srv1904143656=50, srv1632561305=34, srv1703827770=36, srv1742391300=40, srv1911470028=51, srv1507883942=27, srv518132661=75, srv93775933=96, srv1265268468=14, srv817333939=86, srv928189044=94, srv735188718=84, srv1714273460=38, srv1417509809=23, srv247395810=64, srv1130555411=6, srv209640103=61, srv931188025=95, srv824529466=90, srv1331244514=17, srv1049143847=3, srv1957970080=52, srv1108834109=5, srv1317698167=16, srv909343654=93, srv821590934=87, srv177462188=43, srv1818549909=46, srv358169688=68, srv451359680=72, srv822787027=88, srv82400727=89, srv1533019783=28, srv2073213128=58, srv628920660=81, srv1723783444=39, srv1188277185=10, srv1784723395=44, srv1857895624=47, srv2007940100=57, srv1978321636=54, srv1253093159=13, srv1817610745=45, srv653202709=83, srv2134493670=63, srv592990251=78, srv2114705016=62, srv1768003453=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:32,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-23 10:53:32,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-23 10:53:32,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-23 10:53:32,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-23 10:53:32,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-23 10:53:32,896 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:32,896 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1007707984:83 , srv1210499962:83 , srv1338164955:83 , srv1706435144:83 , srv1768003453:83 , srv2001410175:83 , srv209640103:83 , srv517695178:83 , srv817333939:83 , srv88207474:83 , srv1022566736:84 , srv1042485528:84 , srv1072745872:84 , srv1108834109:84 , srv1130555411:84 , srv1135506751:84 , srv1140857428:84 , srv1185640822:84 , srv1188277185:84 , srv1189025658:84 , srv1253093159:84 , srv1265268468:84 , srv1296857680:84 , srv1317698167:84 , srv1331244514:84 , srv1331299282:84 , srv1384518287:84 , srv1385475148:84 , srv1394297707:84 , srv1423482415:84 , srv1448772849:84 , srv1456637231:84 , srv1507883942:84 , srv1533019783:84 , srv1559351532:84 , srv1580184826:84 , srv1592050847:84 , srv1601126092:84 , srv1629225167:84 , srv1632561305:84 , srv168667628:84 , srv1703827770:84 , srv1714273460:84 , srv1742391300:84 , srv177462188:84 , srv1784723395:84 , srv1817610745:84 , srv1818549909:84 , srv1857895624:84 , srv1879436180:84 , srv1895069021:84 , srv1911470028:84 , srv1957970080:84 , srv1978321636:84 , srv198824288:84 , srv2007940100:84 , srv2073213128:84 , srv2087364623:84 , srv2114705016:84 , srv247395810:84 , srv25371597:84 , srv255927781:84 , srv289826033:84 , srv358169688:84 , srv387609646:84 , srv410247104:84 , srv449625734:84 , srv451359680:84 , srv473132547:84 , srv518132661:84 , srv536779855:84 , srv541188727:84 , srv592990251:84 , srv593951847:84 , srv59774876:84 , srv628920660:84 , srv632907398:84 , srv653202709:84 , srv735188718:84 , srv756377171:84 , srv821590934:84 , srv822787027:84 , srv824529466:84 , srv909343654:84 , srv928189044:84 , srv93775933:84 , srv941656270:84 , srv994194064:84 , srv999967856:84 , srv1049143847:85 , srv1723783444:85 , srv1756955675:85 , srv1904143656:85 , srv1975689434:85 , srv2074862807:85 , srv2134493670:85 , srv82400727:85 , srv834937921:85 , srv931188025:85 , srv1417509809:684 } 2024-04-23 10:53:32,933 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=294 (was 276) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=13556 (was 14157) 2024-04-23 10:53:32,944 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=294, ProcessCount=170, AvailableMemoryMB=13555 2024-04-23 10:53:32,945 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-23 10:53:32,945 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-23 10:53:32,958 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007335883:84 , srv1030422909:84 , srv1072459000:84 , srv1079798549:84 , srv1113550221:84 , srv1124014080:84 , srv1200621187:84 , srv1226953558:84 , srv1229242276:84 , srv1344858385:84 , srv1404560324:84 , srv154974585:84 , srv1576224764:84 , srv1605787069:84 , srv1700960709:84 , srv1768710744:84 , srv1886037891:84 , srv1906204699:84 , srv1976053248:84 , srv1989788787:84 , srv202944699:84 , srv2035145661:84 , srv277841204:84 , srv383529647:84 , srv410483173:84 , srv41084853:84 , srv598988023:84 , srv815962117:84 , srv977873821:84 , srv877861883:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-23 10:53:32,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:53:32,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906204699=17, srv1576224764=12, srv1989788787=19, srv598988023=26, srv815962117=27, srv154974585=11, srv877861883=28, srv1404560324=10, srv1700960709=14, srv1229242276=8, srv1605787069=13, srv1886037891=16, srv1072459000=2, srv383529647=23, srv1200621187=6, srv202944699=20, srv1079798549=3, srv1226953558=7, srv1030422909=1, srv1768710744=15, srv977873821=29, srv1344858385=9, srv1113550221=4, srv410483173=24, srv277841204=22, srv1976053248=18, srv1124014080=5, srv1007335883=0, srv2035145661=21, srv41084853=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:53:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:53:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:53:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:53:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:53:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:53:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:53:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-23 10:53:32,999 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1476084283504366, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:53:32,999 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912658152602153, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1476084283504366, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-23 10:55:32,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6732841 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912658152602153 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:55:32,987 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007335883:90 , srv1030422909:90 , srv1072459000:90 , srv1079798549:90 , srv1113550221:90 , srv1124014080:90 , srv1200621187:90 , srv1226953558:90 , srv1229242276:90 , srv1344858385:90 , srv1404560324:90 , srv154974585:90 , srv1576224764:90 , srv1605787069:90 , srv1700960709:90 , srv1768710744:90 , srv1886037891:90 , srv1906204699:90 , srv1976053248:90 , srv1989788787:90 , srv202944699:90 , srv2035145661:90 , srv277841204:90 , srv383529647:90 , srv410483173:90 , srv41084853:90 , srv598988023:90 , srv815962117:90 , srv877861883:90 , srv977873821:90 } 2024-04-23 10:55:32,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-23 10:55:32,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906204699=17, srv1576224764=12, srv1989788787=19, srv598988023=26, srv815962117=27, srv154974585=11, srv877861883=28, srv1404560324=10, srv1700960709=14, srv1229242276=8, srv1605787069=13, srv1886037891=16, srv1072459000=2, srv383529647=23, srv1200621187=6, srv202944699=20, srv1079798549=3, srv1226953558=7, srv1030422909=1, srv1768710744=15, srv977873821=29, srv1344858385=9, srv1113550221=4, srv410483173=24, srv277841204=22, srv1976053248=18, srv1124014080=5, srv1007335883=0, srv2035145661=21, srv41084853=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-23 10:55:32,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-23 10:55:32,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-23 10:55:33,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-23 10:55:33,008 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1007335883:90 , srv1030422909:90 , srv1072459000:90 , srv1079798549:90 , srv1113550221:90 , srv1124014080:90 , srv1200621187:90 , srv1226953558:90 , srv1229242276:90 , srv1344858385:90 , srv1404560324:90 , srv154974585:90 , srv1576224764:90 , srv1605787069:90 , srv1700960709:90 , srv1768710744:90 , srv1886037891:90 , srv1906204699:90 , srv1976053248:90 , srv1989788787:90 , srv202944699:90 , srv2035145661:90 , srv277841204:90 , srv383529647:90 , srv410483173:90 , srv41084853:90 , srv598988023:90 , srv815962117:90 , srv877861883:90 , srv977873821:90 } 2024-04-23 10:55:33,022 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=178 (was 294), ProcessCount=169 (was 170), AvailableMemoryMB=13680 (was 13555) - AvailableMemoryMB LEAK? -