2024-01-09 04:53:17,283 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-09 04:53:17,612 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-09 04:53:17,727 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-09 04:53:17,796 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-09 04:53:17,803 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-09 04:53:17,817 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=167, AvailableMemoryMB=10478 2024-01-09 04:53:17,819 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-09 04:53:17,819 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-09 04:53:17,958 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100781733:84 , srv1009254599:84 , srv1023218743:84 , srv1028807274:84 , srv1031390576:84 , srv1038319938:84 , srv1092607601:84 , srv1111660069:84 , srv1111925538:84 , srv1132092052:84 , srv118351469:84 , srv119248419:84 , srv1196591273:84 , srv1198075296:84 , srv1219925714:84 , srv1220402635:84 , srv1220611718:84 , srv1299273215:84 , srv131523392:84 , srv1363352720:84 , srv1373991906:84 , srv1374531865:84 , srv1398583502:84 , srv1413956544:84 , srv1415899278:84 , srv1427301773:84 , srv1469349563:84 , srv1477908712:84 , srv1484097468:84 , srv1535375928:84 , srv1549834756:84 , srv1582662609:84 , srv1590255535:84 , srv1593567886:84 , srv1629284398:84 , srv1659719926:84 , srv1664512029:84 , srv1670539143:84 , srv1692854638:84 , srv1708701431:84 , srv1709603902:84 , srv1728933463:84 , srv1736332749:84 , srv1748179079:84 , srv1807412550:84 , srv1825941107:84 , srv1851942589:84 , srv1876704434:84 , srv188513886:84 , srv1956948912:84 , srv1968426233:84 , srv2014123228:84 , srv2062953677:84 , srv2074266020:84 , srv211733522:84 , srv232834502:84 , srv250061119:84 , srv257388340:84 , srv275155984:84 , srv300157347:84 , srv313580589:84 , srv362590869:84 , srv365860482:84 , srv392974708:84 , srv403882030:84 , srv405638800:84 , srv415839556:84 , srv418886648:84 , srv447377380:84 , srv471901793:84 , srv479208310:84 , srv482237910:84 , srv483190117:84 , srv488031612:84 , srv515157389:84 , srv556120857:84 , srv595557696:84 , srv59733507:84 , srv605842310:84 , srv607466460:84 , srv620000022:84 , srv641083287:84 , srv698811572:84 , srv71009447:84 , srv722298265:84 , srv726950992:84 , srv751626008:84 , srv754919558:84 , srv774862479:84 , srv808985599:84 , srv822730559:84 , srv827157530:84 , srv856060622:84 , srv899712470:84 , srv906242181:84 , srv928393432:84 , srv931494637:84 , srv934132754:84 , srv951758836:84 , srv886143899:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-09 04:53:17,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:53:17,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1477908712=27, srv482237910=71, srv1413956544=23, srv362590869=61, srv556120857=75, srv607466460=79, srv479208310=70, srv1398583502=22, srv827157530=91, srv313580589=60, srv1111660069=7, srv1692854638=38, srv1736332749=42, srv1219925714=14, srv471901793=69, srv1629284398=34, srv1968426233=50, srv1198075296=13, srv774862479=88, srv1708701431=39, srv899712470=94, srv71009447=83, srv2014123228=51, srv131523392=18, srv1851942589=46, srv1373991906=20, srv515157389=74, srv1031390576=4, srv641083287=81, srv856060622=92, srv1415899278=24, srv1220402635=15, srv2074266020=53, srv934132754=98, srv1659719926=35, srv751626008=86, srv447377380=68, srv250061119=56, srv886143899=93, srv726950992=85, srv1664512029=36, srv1825941107=45, srv1023218743=2, srv1670539143=37, srv488031612=73, srv365860482=62, srv1363352720=19, srv100781733=0, srv1038319938=5, srv1469349563=26, srv822730559=90, srv1593567886=33, srv119248419=11, srv928393432=96, srv2062953677=52, srv118351469=10, srv1299273215=17, srv906242181=95, srv1956948912=49, srv620000022=80, srv188513886=48, srv698811572=82, srv1374531865=21, srv951758836=99, srv1427301773=25, srv275155984=58, srv300157347=59, srv483190117=72, srv1484097468=28, srv1728933463=41, srv1748179079=43, srv392974708=63, srv1535375928=29, srv1876704434=47, srv1111925538=8, srv1590255535=32, srv405638800=65, srv1092607601=6, srv1709603902=40, srv931494637=97, srv403882030=64, srv418886648=67, srv605842310=78, srv722298265=84, srv1009254599=1, srv1132092052=9, srv415839556=66, srv1807412550=44, srv232834502=55, srv1028807274=3, srv1582662609=31, srv1549834756=30, srv754919558=87, srv211733522=54, srv257388340=57, srv808985599=89, srv595557696=76, srv1196591273=12, srv1220611718=16, srv59733507=77} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:53:18,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:53:18,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:53:18,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:53:18,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:53:18,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:53:18,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 04:53:18,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 04:53:18,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 04:53:18,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 04:53:18,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 04:53:18,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 04:53:18,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 04:53:18,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 04:53:18,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 04:53:18,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 04:53:18,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 04:53:18,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 04:53:18,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 04:53:18,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 04:53:18,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 04:53:18,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 04:53:18,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 04:53:18,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 04:53:18,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 04:53:18,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 04:53:18,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 04:53:18,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 04:53:18,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 04:53:18,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 04:53:18,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 04:53:18,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 04:53:18,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 04:53:18,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 04:53:18,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 04:53:18,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 04:53:18,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 04:53:18,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 04:53:18,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 04:53:18,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 04:53:18,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 04:53:18,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 04:53:18,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 04:53:18,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 04:53:18,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 04:53:18,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 04:53:18,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 04:53:18,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 04:53:18,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 04:53:18,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 04:53:18,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 04:53:18,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 04:53:18,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 04:53:18,260 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:18,261 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 04:53:18,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 04:53:18,765 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 649 ms to try 5000 different iterations. Found a solution that moves 4715 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17881304015359625. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03984600610499146, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05739578463856117, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5238888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03984600610499146, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:18,776 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1028807274:81 , srv1196591273:82 , srv1373991906:82 , srv313580589:82 , srv403882030:82 , srv1092607601:83 , srv1111925538:83 , srv1535375928:83 , srv392974708:83 , srv418886648:83 , srv774862479:83 , srv1031390576:84 , srv1220402635:84 , srv1363352720:84 , srv1484097468:84 , srv1593567886:84 , srv1851942589:84 , srv275155984:84 , srv362590869:84 , srv482237910:84 , srv483190117:84 , srv641083287:84 , srv899712470:84 , srv1038319938:85 , srv1469349563:85 , srv1659719926:85 , srv1664512029:85 , srv1692854638:85 , srv188513886:85 , srv250061119:85 , srv365860482:85 , srv479208310:85 , srv698811572:85 , srv726950992:85 , srv808985599:85 , srv1023218743:86 , srv1220611718:86 , srv1415899278:86 , srv1477908712:86 , srv1549834756:86 , srv1629284398:86 , srv1956948912:86 , srv300157347:86 , srv447377380:86 , srv488031612:86 , srv607466460:86 , srv722298265:86 , srv751626008:86 , srv928393432:86 , srv931494637:86 , srv934132754:86 , srv1009254599:87 , srv119248419:87 , srv131523392:87 , srv1582662609:87 , srv1590255535:87 , srv1709603902:87 , srv1736332749:87 , srv1876704434:87 , srv1968426233:87 , srv2014123228:87 , srv471901793:87 , srv595557696:87 , srv59733507:87 , srv620000022:87 , srv71009447:87 , srv906242181:87 , srv1111660069:88 , srv118351469:88 , srv1198075296:88 , srv1219925714:88 , srv1374531865:88 , srv1427301773:88 , srv1748179079:88 , srv1807412550:88 , srv1825941107:88 , srv2062953677:88 , srv2074266020:88 , srv405638800:88 , srv515157389:88 , srv556120857:88 , srv605842310:88 , srv754919558:88 , srv822730559:88 , srv827157530:88 , srv951758836:88 , srv100781733:89 , srv1299273215:89 , srv1398583502:89 , srv1670539143:89 , srv1708701431:89 , srv415839556:89 , srv856060622:89 , srv1413956544:90 , srv257388340:90 , srv1728933463:92 , srv211733522:92 , srv232834502:92 , srv1132092052:102 , srv886143899:444 } 2024-01-09 04:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1477908712=27, srv482237910=71, srv1413956544=23, srv362590869=61, srv556120857=75, srv607466460=79, srv479208310=70, srv1398583502=22, srv827157530=91, srv313580589=60, srv1111660069=7, srv1692854638=38, srv1736332749=42, srv1219925714=14, srv471901793=69, srv1629284398=34, srv1968426233=50, srv1198075296=13, srv774862479=88, srv1708701431=39, srv899712470=94, srv71009447=83, srv2014123228=51, srv131523392=18, srv1851942589=46, srv1373991906=20, srv515157389=74, srv1031390576=4, srv641083287=81, srv856060622=92, srv1415899278=24, srv1220402635=15, srv2074266020=53, srv934132754=98, srv1659719926=35, srv751626008=86, srv447377380=68, srv250061119=56, srv886143899=93, srv726950992=85, srv1664512029=36, srv1825941107=45, srv1023218743=2, srv1670539143=37, srv488031612=73, srv365860482=62, srv1363352720=19, srv100781733=0, srv1038319938=5, srv1469349563=26, srv822730559=90, srv1593567886=33, srv119248419=11, srv928393432=96, srv2062953677=52, srv118351469=10, srv1299273215=17, srv906242181=95, srv1956948912=49, srv620000022=80, srv188513886=48, srv698811572=82, srv1374531865=21, srv951758836=99, srv1427301773=25, srv275155984=58, srv300157347=59, srv483190117=72, srv1484097468=28, srv1728933463=41, srv1748179079=43, srv392974708=63, srv1535375928=29, srv1876704434=47, srv1111925538=8, srv1590255535=32, srv405638800=65, srv1092607601=6, srv1709603902=40, srv931494637=97, srv403882030=64, srv418886648=67, srv605842310=78, srv722298265=84, srv1009254599=1, srv1132092052=9, srv415839556=66, srv1807412550=44, srv232834502=55, srv1028807274=3, srv1582662609=31, srv1549834756=30, srv754919558=87, srv211733522=54, srv257388340=57, srv808985599=89, srv595557696=76, srv1196591273=12, srv1220611718=16, srv59733507=77} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:53:18,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:53:18,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 04:53:18,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 04:53:18,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 04:53:18,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 04:53:18,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 04:53:18,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 04:53:18,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 04:53:18,941 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03984600610499146, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05739578463856117, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03984600610499146, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:18,941 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 04:53:18,941 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1787800250179496, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03984600610499146, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05739578463856117, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03984600610499146, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 04:53:19,215 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 399 ms to try 5000 different iterations. Found a solution that moves 2644 regions; Going from a computed imbalance of 0.1787800250179496 to a new imbalance of 0.014567552028527811. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022981962380033204); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03753636471004274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2937777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022981962380033204); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:19,227 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1132092052:69 , srv1413956544:79 , srv211733522:80 , srv415839556:80 , srv100781733:81 , srv1198075296:81 , srv1398583502:81 , srv257388340:81 , srv405638800:81 , srv754919558:81 , srv1219925714:82 , srv1427301773:82 , srv2062953677:82 , srv2074266020:82 , srv1111660069:83 , srv1582662609:83 , srv1629284398:83 , srv1708701431:83 , srv1709603902:83 , srv1728933463:83 , srv1748179079:83 , srv1825941107:83 , srv2014123228:83 , srv556120857:83 , srv71009447:83 , srv1299273215:84 , srv1374531865:84 , srv1590255535:84 , srv1736332749:84 , srv232834502:84 , srv856060622:84 , srv928393432:84 , srv931494637:84 , srv951758836:84 , srv118351469:85 , srv119248419:85 , srv131523392:85 , srv1415899278:85 , srv1469349563:85 , srv1593567886:85 , srv447377380:85 , srv515157389:85 , srv59733507:85 , srv620000022:85 , srv698811572:85 , srv822730559:85 , srv1009254599:86 , srv1477908712:86 , srv1670539143:86 , srv1692854638:86 , srv1807412550:86 , srv1876704434:86 , srv1956948912:86 , srv250061119:86 , srv300157347:86 , srv479208310:86 , srv595557696:86 , srv605842310:86 , srv827157530:86 , srv934132754:86 , srv1023218743:87 , srv1220402635:87 , srv1220611718:87 , srv1659719926:87 , srv362590869:87 , srv365860482:87 , srv418886648:87 , srv471901793:87 , srv483190117:87 , srv607466460:87 , srv751626008:87 , srv808985599:87 , srv1031390576:88 , srv1111925538:88 , srv1196591273:88 , srv1484097468:88 , srv1549834756:88 , srv1664512029:88 , srv1851942589:88 , srv1968426233:88 , srv403882030:88 , srv774862479:88 , srv906242181:88 , srv1028807274:89 , srv1038319938:89 , srv1092607601:89 , srv1363352720:89 , srv188513886:89 , srv726950992:89 , srv899712470:89 , srv1535375928:90 , srv392974708:90 , srv488031612:90 , srv722298265:90 , srv275155984:91 , srv313580589:91 , srv482237910:91 , srv641083287:91 , srv1373991906:92 , srv886143899:534 } 2024-01-09 04:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1477908712=27, srv482237910=71, srv1413956544=23, srv362590869=61, srv556120857=75, srv607466460=79, srv479208310=70, srv1398583502=22, srv827157530=91, srv313580589=60, srv1111660069=7, srv1692854638=38, srv1736332749=42, srv1219925714=14, srv471901793=69, srv1629284398=34, srv1968426233=50, srv1198075296=13, srv774862479=88, srv1708701431=39, srv899712470=94, srv71009447=83, srv2014123228=51, srv131523392=18, srv1851942589=46, srv1373991906=20, srv515157389=74, srv1031390576=4, srv641083287=81, srv856060622=92, srv1415899278=24, srv1220402635=15, srv2074266020=53, srv934132754=98, srv1659719926=35, srv751626008=86, srv447377380=68, srv250061119=56, srv886143899=93, srv726950992=85, srv1664512029=36, srv1825941107=45, srv1023218743=2, srv1670539143=37, srv488031612=73, srv365860482=62, srv1363352720=19, srv100781733=0, srv1038319938=5, srv1469349563=26, srv822730559=90, srv1593567886=33, srv119248419=11, srv928393432=96, srv2062953677=52, srv118351469=10, srv1299273215=17, srv906242181=95, srv1956948912=49, srv620000022=80, srv188513886=48, srv698811572=82, srv1374531865=21, srv951758836=99, srv1427301773=25, srv275155984=58, srv300157347=59, srv483190117=72, srv1484097468=28, srv1728933463=41, srv1748179079=43, srv392974708=63, srv1535375928=29, srv1876704434=47, srv1111925538=8, srv1590255535=32, srv405638800=65, srv1092607601=6, srv1709603902=40, srv931494637=97, srv403882030=64, srv418886648=67, srv605842310=78, srv722298265=84, srv1009254599=1, srv1132092052=9, srv415839556=66, srv1807412550=44, srv232834502=55, srv1028807274=3, srv1582662609=31, srv1549834756=30, srv754919558=87, srv211733522=54, srv257388340=57, srv808985599=89, srv595557696=76, srv1196591273=12, srv1220611718=16, srv59733507=77} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:53:19,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:53:19,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:53:19,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:53:19,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:53:19,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:53:19,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:53:19,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 04:53:19,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 04:53:19,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 04:53:19,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 04:53:19,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 04:53:19,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 04:53:19,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 04:53:19,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 04:53:19,376 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022981962380033204); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03753636471004274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022981962380033204); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:19,376 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 04:53:19,376 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014549038344826916, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022981962380033204); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03753636471004274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022981962380033204); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 04:53:19,728 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations. Found a solution that moves 1099 regions; Going from a computed imbalance of 0.014549038344826916 to a new imbalance of 6.432043632232337E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005090526057598092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007400670336975435); MoveCostFunction : (multiplier=7.0, imbalance=0.12211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005090526057598092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:19,729 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv232834502:81 , srv1728933463:82 , srv482237910:82 , srv488031612:82 , srv722298265:82 , srv1038319938:83 , srv1299273215:83 , srv1373991906:83 , srv1670539143:83 , srv1807412550:83 , srv188513886:83 , srv1968426233:83 , srv275155984:83 , srv471901793:83 , srv605842310:83 , srv641083287:83 , srv906242181:83 , srv1009254599:84 , srv1535375928:84 , srv1549834756:84 , srv313580589:84 , srv515157389:84 , srv607466460:84 , srv726950992:84 , srv856060622:84 , srv1023218743:85 , srv1031390576:85 , srv1092607601:85 , srv118351469:85 , srv119248419:85 , srv1220611718:85 , srv1363352720:85 , srv1659719926:85 , srv1664512029:85 , srv1851942589:85 , srv1876704434:85 , srv211733522:85 , srv392974708:85 , srv595557696:85 , srv59733507:85 , srv620000022:85 , srv751626008:85 , srv822730559:85 , srv827157530:85 , srv899712470:85 , srv934132754:85 , srv1111925538:86 , srv1132092052:86 , srv1220402635:86 , srv131523392:86 , srv1374531865:86 , srv1477908712:86 , srv1484097468:86 , srv1708701431:86 , srv1825941107:86 , srv1956948912:86 , srv257388340:86 , srv300157347:86 , srv362590869:86 , srv365860482:86 , srv556120857:86 , srv808985599:86 , srv951758836:86 , srv100781733:87 , srv1028807274:87 , srv1196591273:87 , srv1398583502:87 , srv1415899278:87 , srv1590255535:87 , srv1692854638:87 , srv1709603902:87 , srv1736332749:87 , srv1748179079:87 , srv2062953677:87 , srv250061119:87 , srv403882030:87 , srv418886648:87 , srv447377380:87 , srv483190117:87 , srv698811572:87 , srv71009447:87 , srv774862479:87 , srv928393432:87 , srv931494637:87 , srv1111660069:88 , srv1198075296:88 , srv1219925714:88 , srv1427301773:88 , srv1469349563:88 , srv1582662609:88 , srv1593567886:88 , srv2014123228:88 , srv2074266020:88 , srv415839556:88 , srv479208310:88 , srv1413956544:89 , srv1629284398:89 , srv405638800:89 , srv754919558:89 , srv886143899:525 } 2024-01-09 04:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1477908712=27, srv482237910=71, srv1413956544=23, srv362590869=61, srv556120857=75, srv607466460=79, srv479208310=70, srv1398583502=22, srv827157530=91, srv313580589=60, srv1111660069=7, srv1692854638=38, srv1736332749=42, srv1219925714=14, srv471901793=69, srv1629284398=34, srv1968426233=50, srv1198075296=13, srv774862479=88, srv1708701431=39, srv899712470=94, srv71009447=83, srv2014123228=51, srv131523392=18, srv1851942589=46, srv1373991906=20, srv515157389=74, srv1031390576=4, srv641083287=81, srv856060622=92, srv1415899278=24, srv1220402635=15, srv2074266020=53, srv934132754=98, srv1659719926=35, srv751626008=86, srv447377380=68, srv250061119=56, srv886143899=93, srv726950992=85, srv1664512029=36, srv1825941107=45, srv1023218743=2, srv1670539143=37, srv488031612=73, srv365860482=62, srv1363352720=19, srv100781733=0, srv1038319938=5, srv1469349563=26, srv822730559=90, srv1593567886=33, srv119248419=11, srv928393432=96, srv2062953677=52, srv118351469=10, srv1299273215=17, srv906242181=95, srv1956948912=49, srv620000022=80, srv188513886=48, srv698811572=82, srv1374531865=21, srv951758836=99, srv1427301773=25, srv275155984=58, srv300157347=59, srv483190117=72, srv1484097468=28, srv1728933463=41, srv1748179079=43, srv392974708=63, srv1535375928=29, srv1876704434=47, srv1111925538=8, srv1590255535=32, srv405638800=65, srv1092607601=6, srv1709603902=40, srv931494637=97, srv403882030=64, srv418886648=67, srv605842310=78, srv722298265=84, srv1009254599=1, srv1132092052=9, srv415839556=66, srv1807412550=44, srv232834502=55, srv1028807274=3, srv1582662609=31, srv1549834756=30, srv754919558=87, srv211733522=54, srv257388340=57, srv808985599=89, srv595557696=76, srv1196591273=12, srv1220611718=16, srv59733507=77} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:53:19,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:53:19,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:53:19,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 04:53:19,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 04:53:19,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 04:53:19,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 04:53:19,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 04:53:19,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 04:53:19,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 04:53:19,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 04:53:19,859 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005090526057598092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007400670336975435); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005090526057598092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:19,859 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 04:53:19,860 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.355090007469535E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005090526057598092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007400670336975435); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005090526057598092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 04:53:20,108 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 343 ms to try 5000 different iterations. Found a solution that moves 240 regions; Going from a computed imbalance of 6.355090007469535E-4 to a new imbalance of 1.9025712164535136E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.02666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:20,109 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1111660069:83 , srv1670539143:83 , srv827157530:83 , srv1023218743:84 , srv1132092052:84 , srv118351469:84 , srv1219925714:84 , srv1220611718:84 , srv131523392:84 , srv1374531865:84 , srv1413956544:84 , srv1415899278:84 , srv1427301773:84 , srv1469349563:84 , srv1477908712:84 , srv1484097468:84 , srv1549834756:84 , srv1582662609:84 , srv1590255535:84 , srv1629284398:84 , srv1664512029:84 , srv1692854638:84 , srv1708701431:84 , srv1736332749:84 , srv1748179079:84 , srv1876704434:84 , srv1956948912:84 , srv1968426233:84 , srv2014123228:84 , srv2074266020:84 , srv250061119:84 , srv257388340:84 , srv275155984:84 , srv300157347:84 , srv365860482:84 , srv392974708:84 , srv405638800:84 , srv447377380:84 , srv479208310:84 , srv483190117:84 , srv488031612:84 , srv595557696:84 , srv59733507:84 , srv641083287:84 , srv722298265:84 , srv726950992:84 , srv751626008:84 , srv754919558:84 , srv774862479:84 , srv808985599:84 , srv822730559:84 , srv899712470:84 , srv931494637:84 , srv951758836:84 , srv100781733:85 , srv1009254599:85 , srv1028807274:85 , srv1031390576:85 , srv1038319938:85 , srv1092607601:85 , srv1111925538:85 , srv1196591273:85 , srv1198075296:85 , srv1220402635:85 , srv1363352720:85 , srv1373991906:85 , srv1398583502:85 , srv1535375928:85 , srv1593567886:85 , srv1659719926:85 , srv1709603902:85 , srv1728933463:85 , srv1807412550:85 , srv1825941107:85 , srv1851942589:85 , srv188513886:85 , srv2062953677:85 , srv211733522:85 , srv232834502:85 , srv362590869:85 , srv403882030:85 , srv415839556:85 , srv418886648:85 , srv471901793:85 , srv482237910:85 , srv515157389:85 , srv556120857:85 , srv605842310:85 , srv607466460:85 , srv620000022:85 , srv698811572:85 , srv71009447:85 , srv906242181:85 , srv928393432:85 , srv934132754:85 , srv119248419:86 , srv1299273215:86 , srv313580589:86 , srv856060622:86 , srv886143899:638 } 2024-01-09 04:53:20,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:53:20,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1477908712=27, srv482237910=71, srv1413956544=23, srv362590869=61, srv556120857=75, srv607466460=79, srv479208310=70, srv1398583502=22, srv827157530=91, srv313580589=60, srv1111660069=7, srv1692854638=38, srv1736332749=42, srv1219925714=14, srv471901793=69, srv1629284398=34, srv1968426233=50, srv1198075296=13, srv774862479=88, srv1708701431=39, srv899712470=94, srv71009447=83, srv2014123228=51, srv131523392=18, srv1851942589=46, srv1373991906=20, srv515157389=74, srv1031390576=4, srv641083287=81, srv856060622=92, srv1415899278=24, srv1220402635=15, srv2074266020=53, srv934132754=98, srv1659719926=35, srv751626008=86, srv447377380=68, srv250061119=56, srv886143899=93, srv726950992=85, srv1664512029=36, srv1825941107=45, srv1023218743=2, srv1670539143=37, srv488031612=73, srv365860482=62, srv1363352720=19, srv100781733=0, srv1038319938=5, srv1469349563=26, srv822730559=90, srv1593567886=33, srv119248419=11, srv928393432=96, srv2062953677=52, srv118351469=10, srv1299273215=17, srv906242181=95, srv1956948912=49, srv620000022=80, srv188513886=48, srv698811572=82, srv1374531865=21, srv951758836=99, srv1427301773=25, srv275155984=58, srv300157347=59, srv483190117=72, srv1484097468=28, srv1728933463=41, srv1748179079=43, srv392974708=63, srv1535375928=29, srv1876704434=47, srv1111925538=8, srv1590255535=32, srv405638800=65, srv1092607601=6, srv1709603902=40, srv931494637=97, srv403882030=64, srv418886648=67, srv605842310=78, srv722298265=84, srv1009254599=1, srv1132092052=9, srv415839556=66, srv1807412550=44, srv232834502=55, srv1028807274=3, srv1582662609=31, srv1549834756=30, srv754919558=87, srv211733522=54, srv257388340=57, srv808985599=89, srv595557696=76, srv1196591273=12, srv1220611718=16, srv59733507=77} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 04:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 04:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 04:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 04:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 04:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 04:53:20,232 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.734519624614819E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:20,233 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1111660069:83 , srv1670539143:83 , srv827157530:83 , srv1023218743:84 , srv1132092052:84 , srv118351469:84 , srv1219925714:84 , srv1220611718:84 , srv131523392:84 , srv1374531865:84 , srv1413956544:84 , srv1415899278:84 , srv1427301773:84 , srv1469349563:84 , srv1477908712:84 , srv1484097468:84 , srv1549834756:84 , srv1582662609:84 , srv1590255535:84 , srv1629284398:84 , srv1664512029:84 , srv1692854638:84 , srv1708701431:84 , srv1736332749:84 , srv1748179079:84 , srv1876704434:84 , srv1956948912:84 , srv1968426233:84 , srv2014123228:84 , srv2074266020:84 , srv250061119:84 , srv257388340:84 , srv275155984:84 , srv300157347:84 , srv365860482:84 , srv392974708:84 , srv405638800:84 , srv447377380:84 , srv479208310:84 , srv483190117:84 , srv488031612:84 , srv595557696:84 , srv59733507:84 , srv641083287:84 , srv722298265:84 , srv726950992:84 , srv751626008:84 , srv754919558:84 , srv774862479:84 , srv808985599:84 , srv822730559:84 , srv899712470:84 , srv931494637:84 , srv951758836:84 , srv100781733:85 , srv1009254599:85 , srv1028807274:85 , srv1031390576:85 , srv1038319938:85 , srv1092607601:85 , srv1111925538:85 , srv1196591273:85 , srv1198075296:85 , srv1220402635:85 , srv1363352720:85 , srv1373991906:85 , srv1398583502:85 , srv1535375928:85 , srv1593567886:85 , srv1659719926:85 , srv1709603902:85 , srv1728933463:85 , srv1807412550:85 , srv1825941107:85 , srv1851942589:85 , srv188513886:85 , srv2062953677:85 , srv211733522:85 , srv232834502:85 , srv362590869:85 , srv403882030:85 , srv415839556:85 , srv418886648:85 , srv471901793:85 , srv482237910:85 , srv515157389:85 , srv556120857:85 , srv605842310:85 , srv607466460:85 , srv620000022:85 , srv698811572:85 , srv71009447:85 , srv906242181:85 , srv928393432:85 , srv934132754:85 , srv119248419:86 , srv1299273215:86 , srv313580589:86 , srv856060622:86 , srv886143899:638 } 2024-01-09 04:53:20,258 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 337), ProcessCount=167 (was 167), AvailableMemoryMB=9891 (was 10478) 2024-01-09 04:53:20,266 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=167, AvailableMemoryMB=9889 2024-01-09 04:53:20,267 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-09 04:53:20,267 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-09 04:53:20,283 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1314739940:84 , srv1411383876:84 , srv1416492990:84 , srv1430472459:84 , srv1500248575:84 , srv1568914937:84 , srv1611128839:84 , srv1672478157:84 , srv1729140372:84 , srv1739711841:84 , srv1798232194:84 , srv1819789580:84 , srv1859652584:84 , srv1894163866:84 , srv1924008525:84 , srv2033570219:84 , srv214358200:84 , srv24209708:84 , srv318432246:84 , srv320461849:84 , srv330211895:84 , srv414379795:84 , srv502770810:84 , srv514714781:84 , srv548319345:84 , srv609995718:84 , srv736706125:84 , srv821526243:84 , srv823584826:84 , srv398990157:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-09 04:53:20,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:53:20,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314739940=0, srv1611128839=6, srv1672478157=7, srv548319345=25, srv1798232194=10, srv24209708=17, srv1411383876=1, srv1729140372=8, srv1430472459=3, srv1568914937=5, srv1416492990=2, srv320461849=19, srv214358200=16, srv398990157=21, srv823584826=29, srv1739711841=9, srv1819789580=11, srv736706125=27, srv1500248575=4, srv1859652584=12, srv1924008525=14, srv414379795=22, srv502770810=23, srv2033570219=15, srv821526243=28, srv514714781=24, srv609995718=26, srv1894163866=13, srv318432246=18, srv330211895=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-09 04:53:20,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0768903403900427, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:53:20,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991243532221615, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0768903403900427, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-09 04:55:20,357 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120052 ms to try 5578300 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.991243532221615 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:55:20,366 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1314739940:90 , srv1411383876:90 , srv1416492990:90 , srv1430472459:90 , srv1500248575:90 , srv1568914937:90 , srv1611128839:90 , srv1672478157:90 , srv1729140372:90 , srv1739711841:90 , srv1798232194:90 , srv1819789580:90 , srv1859652584:90 , srv1894163866:90 , srv1924008525:90 , srv2033570219:90 , srv214358200:90 , srv24209708:90 , srv318432246:90 , srv320461849:90 , srv330211895:90 , srv398990157:90 , srv414379795:90 , srv502770810:90 , srv514714781:90 , srv548319345:90 , srv609995718:90 , srv736706125:90 , srv821526243:90 , srv823584826:90 } 2024-01-09 04:55:20,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 04:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314739940=0, srv1611128839=6, srv1672478157=7, srv548319345=25, srv1798232194=10, srv24209708=17, srv1411383876=1, srv1729140372=8, srv1430472459=3, srv1568914937=5, srv1416492990=2, srv320461849=19, srv214358200=16, srv398990157=21, srv823584826=29, srv1739711841=9, srv1819789580=11, srv736706125=27, srv1500248575=4, srv1859652584=12, srv1924008525=14, srv414379795=22, srv502770810=23, srv2033570219=15, srv821526243=28, srv514714781=24, srv609995718=26, srv1894163866=13, srv318432246=18, srv330211895=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 04:55:20,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 04:55:20,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 04:55:20,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 04:55:20,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 04:55:20,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-09 04:55:20,540 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 04:55:20,540 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1314739940:90 , srv1411383876:90 , srv1416492990:90 , srv1430472459:90 , srv1500248575:90 , srv1568914937:90 , srv1611128839:90 , srv1672478157:90 , srv1729140372:90 , srv1739711841:90 , srv1798232194:90 , srv1819789580:90 , srv1859652584:90 , srv1894163866:90 , srv1924008525:90 , srv2033570219:90 , srv214358200:90 , srv24209708:90 , srv318432246:90 , srv320461849:90 , srv330211895:90 , srv398990157:90 , srv414379795:90 , srv502770810:90 , srv514714781:90 , srv548319345:90 , srv609995718:90 , srv736706125:90 , srv821526243:90 , srv823584826:90 } 2024-01-09 04:55:20,702 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=216 (was 337), ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=9303 (was 9889)