2024-04-14 04:53:15,301 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-14 04:53:15,619 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-14 04:53:15,738 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-14 04:53:15,806 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-14 04:53:15,813 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-14 04:53:15,832 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=319, ProcessCount=169, AvailableMemoryMB=8644 2024-04-14 04:53:15,834 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-14 04:53:15,834 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-14 04:53:15,966 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004982478:84 , srv1034717674:84 , srv1045930430:84 , srv1070687915:84 , srv1123675532:84 , srv1132968419:84 , srv1189778139:84 , srv1201361808:84 , srv120233800:84 , srv1202808556:84 , srv1206392108:84 , srv1218889633:84 , srv1225898884:84 , srv1242303034:84 , srv1267317716:84 , srv128714631:84 , srv1287699103:84 , srv1376334674:84 , srv1378973372:84 , srv1381737025:84 , srv1383432747:84 , srv1397560139:84 , srv1412870643:84 , srv1437517826:84 , srv1446170587:84 , srv1506852492:84 , srv151240652:84 , srv1513309329:84 , srv1567673943:84 , srv1626812334:84 , srv1629823077:84 , srv1643175525:84 , srv1649494316:84 , srv1669055882:84 , srv1717170115:84 , srv1743540914:84 , srv1750195808:84 , srv1755624780:84 , srv177191316:84 , srv1802476697:84 , srv1808301048:84 , srv1833797085:84 , srv1885150818:84 , srv1897015294:84 , srv1910300667:84 , srv1910642058:84 , srv1950579888:84 , srv2011116593:84 , srv2045747975:84 , srv2062912588:84 , srv2068015187:84 , srv2125449227:84 , srv213212694:84 , srv214885110:84 , srv235971114:84 , srv257256088:84 , srv267492269:84 , srv292196539:84 , srv326781486:84 , srv332722509:84 , srv336140879:84 , srv351972372:84 , srv354041766:84 , srv363038237:84 , srv365358091:84 , srv402684271:84 , srv434184021:84 , srv444143122:84 , srv483744120:84 , srv484925100:84 , srv557588707:84 , srv60546987:84 , srv63346045:84 , srv643679448:84 , srv661885190:84 , srv661908597:84 , srv666837819:84 , srv682453089:84 , srv693142401:84 , srv699700503:84 , srv716121343:84 , srv726303018:84 , srv807633389:84 , srv820267177:84 , srv823075226:84 , srv825230235:84 , srv826344366:84 , srv849573990:84 , srv879968594:84 , srv904303876:84 , srv916666736:84 , srv928160621:84 , srv930473294:84 , srv930777275:84 , srv936715734:84 , srv946374141:84 , srv947564682:84 , srv9575988:84 , srv989261164:84 , srv981979757:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-14 04:53:15,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv726303018=81, srv1750195808=36, srv1833797085=41, srv1206392108=10, srv1397560139=21, srv2045747975=48, srv716121343=80, srv1446170587=24, srv1885150818=42, srv1755624780=37, srv444143122=67, srv930777275=93, srv989261164=99, srv1412870643=22, srv1267317716=14, srv213212694=52, srv1004982478=0, srv363038237=63, srv826344366=86, srv2125449227=51, srv1034717674=1, srv1897015294=43, srv1045930430=2, srv120233800=8, srv1383432747=20, srv666837819=76, srv1910642058=45, srv365358091=64, srv682453089=77, srv60546987=71, srv820267177=83, srv1717170115=34, srv1381737025=19, srv936715734=94, srv946374141=95, srv661885190=74, srv257256088=55, srv1218889633=11, srv1202808556=9, srv904303876=89, srv351972372=61, srv849573990=87, srv1132968419=5, srv1743540914=35, srv332722509=59, srv699700503=79, srv879968594=88, srv661908597=75, srv128714631=15, srv326781486=58, srv1070687915=3, srv823075226=84, srv928160621=91, srv151240652=26, srv1225898884=12, srv825230235=85, srv235971114=54, srv1643175525=31, srv1189778139=6, srv1287699103=16, srv693142401=78, srv177191316=38, srv643679448=73, srv1123675532=4, srv1950579888=46, srv1802476697=39, srv807633389=82, srv916666736=90, srv354041766=62, srv1626812334=29, srv1201361808=7, srv1376334674=17, srv63346045=72, srv484925100=69, srv2068015187=50, srv981979757=98, srv1669055882=33, srv214885110=53, srv1506852492=25, srv292196539=57, srv402684271=65, srv336140879=60, srv1567673943=28, srv1910300667=44, srv1437517826=23, srv2062912588=49, srv1649494316=32, srv1629823077=30, srv1378973372=18, srv930473294=92, srv1513309329=27, srv483744120=68, srv1808301048=40, srv2011116593=47, srv267492269=56, srv434184021=66, srv947564682=96, srv557588707=70, srv1242303034=13, srv9575988=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:16,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:16,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:16,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:16,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:16,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:16,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:16,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:16,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-14 04:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-14 04:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-14 04:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-14 04:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-14 04:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-14 04:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-14 04:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-14 04:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-14 04:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-14 04:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-14 04:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-14 04:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-14 04:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-14 04:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-14 04:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-14 04:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-14 04:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-14 04:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-14 04:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-14 04:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-14 04:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-14 04:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-14 04:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-14 04:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-14 04:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-14 04:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-14 04:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-14 04:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-14 04:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-14 04:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-14 04:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-14 04:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-14 04:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-14 04:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-14 04:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-14 04:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-14 04:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-14 04:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-14 04:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-14 04:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-14 04:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-14 04:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-14 04:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-14 04:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-14 04:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-14 04:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-14 04:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-14 04:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-14 04:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-14 04:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-14 04:53:16,498 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:16,499 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-14 04:53:16,499 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-14 04:53:16,841 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 721 ms to try 5000 different iterations. Found a solution that moves 4645 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18380722260077392. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04182636809434434, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05986329356915847, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5161111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04182636809434434, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17883333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:16,862 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1034717674:82 , srv1383432747:82 , srv1802476697:82 , srv2011116593:82 , srv904303876:82 , srv928160621:82 , srv1567673943:83 , srv402684271:83 , srv661885190:83 , srv826344366:83 , srv1004982478:84 , srv1045930430:84 , srv1132968419:84 , srv1202808556:84 , srv1225898884:84 , srv151240652:84 , srv1897015294:84 , srv2062912588:84 , srv63346045:84 , srv849573990:84 , srv1070687915:85 , srv1412870643:85 , srv1437517826:85 , srv1513309329:85 , srv1743540914:85 , srv1750195808:85 , srv1833797085:85 , srv213212694:85 , srv257256088:85 , srv267492269:85 , srv332722509:85 , srv363038237:85 , srv60546987:85 , srv643679448:85 , srv661908597:85 , srv682453089:85 , srv916666736:85 , srv930473294:85 , srv930777275:85 , srv947564682:85 , srv120233800:86 , srv1218889633:86 , srv128714631:86 , srv1287699103:86 , srv1446170587:86 , srv1626812334:86 , srv1808301048:86 , srv1910300667:86 , srv2045747975:86 , srv2125449227:86 , srv326781486:86 , srv354041766:86 , srv483744120:86 , srv484925100:86 , srv936715734:86 , srv946374141:86 , srv989261164:86 , srv1267317716:87 , srv1506852492:87 , srv1649494316:87 , srv1755624780:87 , srv177191316:87 , srv1950579888:87 , srv336140879:87 , srv666837819:87 , srv716121343:87 , srv807633389:87 , srv820267177:87 , srv823075226:87 , srv1123675532:88 , srv1189778139:88 , srv1206392108:88 , srv1242303034:88 , srv1376334674:88 , srv1643175525:88 , srv1669055882:88 , srv1885150818:88 , srv1910642058:88 , srv2068015187:88 , srv214885110:88 , srv235971114:88 , srv434184021:88 , srv444143122:88 , srv557588707:88 , srv699700503:88 , srv879968594:88 , srv1378973372:89 , srv1381737025:89 , srv1397560139:89 , srv365358091:89 , srv9575988:89 , srv292196539:90 , srv693142401:90 , srv726303018:90 , srv1201361808:91 , srv351972372:91 , srv825230235:91 , srv1629823077:92 , srv1717170115:92 , srv981979757:462 } 2024-04-14 04:53:16,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:16,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv726303018=81, srv1750195808=36, srv1833797085=41, srv1206392108=10, srv1397560139=21, srv2045747975=48, srv716121343=80, srv1446170587=24, srv1885150818=42, srv1755624780=37, srv444143122=67, srv930777275=93, srv989261164=99, srv1412870643=22, srv1267317716=14, srv213212694=52, srv1004982478=0, srv363038237=63, srv826344366=86, srv2125449227=51, srv1034717674=1, srv1897015294=43, srv1045930430=2, srv120233800=8, srv1383432747=20, srv666837819=76, srv1910642058=45, srv365358091=64, srv682453089=77, srv60546987=71, srv820267177=83, srv1717170115=34, srv1381737025=19, srv936715734=94, srv946374141=95, srv661885190=74, srv257256088=55, srv1218889633=11, srv1202808556=9, srv904303876=89, srv351972372=61, srv849573990=87, srv1132968419=5, srv1743540914=35, srv332722509=59, srv699700503=79, srv879968594=88, srv661908597=75, srv128714631=15, srv326781486=58, srv1070687915=3, srv823075226=84, srv928160621=91, srv151240652=26, srv1225898884=12, srv825230235=85, srv235971114=54, srv1643175525=31, srv1189778139=6, srv1287699103=16, srv693142401=78, srv177191316=38, srv643679448=73, srv1123675532=4, srv1950579888=46, srv1802476697=39, srv807633389=82, srv916666736=90, srv354041766=62, srv1626812334=29, srv1201361808=7, srv1376334674=17, srv63346045=72, srv484925100=69, srv2068015187=50, srv981979757=98, srv1669055882=33, srv214885110=53, srv1506852492=25, srv292196539=57, srv402684271=65, srv336140879=60, srv1567673943=28, srv1910300667=44, srv1437517826=23, srv2062912588=49, srv1649494316=32, srv1629823077=30, srv1378973372=18, srv930473294=92, srv1513309329=27, srv483744120=68, srv1808301048=40, srv2011116593=47, srv267492269=56, srv434184021=66, srv947564682=96, srv557588707=70, srv1242303034=13, srv9575988=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:16,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:16,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-14 04:53:16,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-14 04:53:16,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-14 04:53:16,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-14 04:53:16,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-14 04:53:16,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-14 04:53:16,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-14 04:53:16,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-14 04:53:16,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-14 04:53:16,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-14 04:53:16,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-14 04:53:16,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-14 04:53:16,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-14 04:53:16,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-14 04:53:16,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-14 04:53:16,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-14 04:53:16,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-14 04:53:16,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-14 04:53:16,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-14 04:53:17,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04182636809434434, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05986329356915847, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04182636809434434, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17883333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:17,061 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-14 04:53:17,061 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1837746976156035, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04182636809434434, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05986329356915847, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04182636809434434, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17883333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-14 04:53:17,336 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 411 ms to try 5000 different iterations. Found a solution that moves 2658 regions; Going from a computed imbalance of 0.1837746976156035 to a new imbalance of 0.01577414405955839. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021384898399012836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035606024786857836, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29533333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021384898399012836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:17,347 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1629823077:78 , srv1717170115:80 , srv292196539:80 , srv9575988:81 , srv1378973372:82 , srv1885150818:82 , srv351972372:82 , srv693142401:82 , srv1123675532:83 , srv1189778139:83 , srv1376334674:83 , srv1397560139:83 , srv177191316:83 , srv2068015187:83 , srv235971114:83 , srv336140879:83 , srv444143122:83 , srv1201361808:84 , srv1206392108:84 , srv1381737025:84 , srv1506852492:84 , srv1643175525:84 , srv1669055882:84 , srv214885110:84 , srv365358091:84 , srv483744120:84 , srv557588707:84 , srv699700503:84 , srv726303018:84 , srv879968594:84 , srv1267317716:85 , srv1287699103:85 , srv1446170587:85 , srv1649494316:85 , srv1833797085:85 , srv1910642058:85 , srv484925100:85 , srv661908597:85 , srv666837819:85 , srv820267177:85 , srv823075226:85 , srv946374141:85 , srv1218889633:86 , srv1242303034:86 , srv1437517826:86 , srv1755624780:86 , srv1897015294:86 , srv1910300667:86 , srv2062912588:86 , srv2125449227:86 , srv326781486:86 , srv332722509:86 , srv354041766:86 , srv434184021:86 , srv60546987:86 , srv682453089:86 , srv807633389:86 , srv825230235:86 , srv936715734:86 , srv1070687915:87 , srv1202808556:87 , srv128714631:87 , srv151240652:87 , srv1513309329:87 , srv1743540914:87 , srv1750195808:87 , srv1808301048:87 , srv1950579888:87 , srv2045747975:87 , srv716121343:87 , srv849573990:87 , srv916666736:87 , srv1004982478:88 , srv1034717674:88 , srv1132968419:88 , srv120233800:88 , srv1225898884:88 , srv1412870643:88 , srv1626812334:88 , srv1802476697:88 , srv257256088:88 , srv363038237:88 , srv643679448:88 , srv661885190:88 , srv930777275:88 , srv947564682:88 , srv989261164:88 , srv1383432747:89 , srv1567673943:89 , srv826344366:89 , srv928160621:89 , srv1045930430:90 , srv2011116593:90 , srv267492269:90 , srv402684271:90 , srv63346045:90 , srv930473294:90 , srv904303876:91 , srv213212694:92 , srv981979757:502 } 2024-04-14 04:53:17,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:17,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv726303018=81, srv1750195808=36, srv1833797085=41, srv1206392108=10, srv1397560139=21, srv2045747975=48, srv716121343=80, srv1446170587=24, srv1885150818=42, srv1755624780=37, srv444143122=67, srv930777275=93, srv989261164=99, srv1412870643=22, srv1267317716=14, srv213212694=52, srv1004982478=0, srv363038237=63, srv826344366=86, srv2125449227=51, srv1034717674=1, srv1897015294=43, srv1045930430=2, srv120233800=8, srv1383432747=20, srv666837819=76, srv1910642058=45, srv365358091=64, srv682453089=77, srv60546987=71, srv820267177=83, srv1717170115=34, srv1381737025=19, srv936715734=94, srv946374141=95, srv661885190=74, srv257256088=55, srv1218889633=11, srv1202808556=9, srv904303876=89, srv351972372=61, srv849573990=87, srv1132968419=5, srv1743540914=35, srv332722509=59, srv699700503=79, srv879968594=88, srv661908597=75, srv128714631=15, srv326781486=58, srv1070687915=3, srv823075226=84, srv928160621=91, srv151240652=26, srv1225898884=12, srv825230235=85, srv235971114=54, srv1643175525=31, srv1189778139=6, srv1287699103=16, srv693142401=78, srv177191316=38, srv643679448=73, srv1123675532=4, srv1950579888=46, srv1802476697=39, srv807633389=82, srv916666736=90, srv354041766=62, srv1626812334=29, srv1201361808=7, srv1376334674=17, srv63346045=72, srv484925100=69, srv2068015187=50, srv981979757=98, srv1669055882=33, srv214885110=53, srv1506852492=25, srv292196539=57, srv402684271=65, srv336140879=60, srv1567673943=28, srv1910300667=44, srv1437517826=23, srv2062912588=49, srv1649494316=32, srv1629823077=30, srv1378973372=18, srv930473294=92, srv1513309329=27, srv483744120=68, srv1808301048=40, srv2011116593=47, srv267492269=56, srv434184021=66, srv947564682=96, srv557588707=70, srv1242303034=13, srv9575988=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-14 04:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-14 04:53:17,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-14 04:53:17,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-14 04:53:17,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-14 04:53:17,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-14 04:53:17,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-14 04:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-14 04:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-14 04:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-14 04:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-14 04:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-14 04:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-14 04:53:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-14 04:53:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-14 04:53:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-14 04:53:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-14 04:53:17,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-14 04:53:17,518 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021384898399012836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035606024786857836, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021384898399012836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:17,518 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-14 04:53:17,518 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015755532345762255, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021384898399012836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035606024786857836, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021384898399012836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-14 04:53:17,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 358 ms to try 5000 different iterations. Found a solution that moves 1006 regions; Going from a computed imbalance of 0.015755532345762255 to a new imbalance of 7.592963208207659E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010938046122328536); MoveCostFunction : (multiplier=7.0, imbalance=0.11177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:17,765 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv213212694:81 , srv825230235:81 , srv1201361808:82 , srv267492269:82 , srv1045930430:83 , srv120233800:83 , srv1242303034:83 , srv1626812334:83 , srv1950579888:83 , srv63346045:83 , srv930473294:83 , srv989261164:83 , srv1412870643:84 , srv214885110:84 , srv257256088:84 , srv402684271:84 , srv434184021:84 , srv557588707:84 , srv643679448:84 , srv716121343:84 , srv726303018:84 , srv807633389:84 , srv904303876:84 , srv1004982478:85 , srv1070687915:85 , srv1132968419:85 , srv1202808556:85 , srv1206392108:85 , srv1218889633:85 , srv1225898884:85 , srv1267317716:85 , srv128714631:85 , srv1378973372:85 , srv1381737025:85 , srv1397560139:85 , srv151240652:85 , srv1649494316:85 , srv1717170115:85 , srv1743540914:85 , srv1750195808:85 , srv1755624780:85 , srv1808301048:85 , srv1910300667:85 , srv1910642058:85 , srv2011116593:85 , srv2045747975:85 , srv326781486:85 , srv351972372:85 , srv354041766:85 , srv363038237:85 , srv365358091:85 , srv484925100:85 , srv666837819:85 , srv693142401:85 , srv820267177:85 , srv826344366:85 , srv879968594:85 , srv916666736:85 , srv930777275:85 , srv947564682:85 , srv1123675532:86 , srv1376334674:86 , srv1437517826:86 , srv1446170587:86 , srv1506852492:86 , srv1513309329:86 , srv1567673943:86 , srv1669055882:86 , srv1802476697:86 , srv2068015187:86 , srv235971114:86 , srv661885190:86 , srv682453089:86 , srv699700503:86 , srv823075226:86 , srv936715734:86 , srv9575988:86 , srv1034717674:87 , srv1189778139:87 , srv1287699103:87 , srv1383432747:87 , srv1629823077:87 , srv1643175525:87 , srv2125449227:87 , srv292196539:87 , srv332722509:87 , srv336140879:87 , srv444143122:87 , srv483744120:87 , srv60546987:87 , srv849573990:87 , srv928160621:87 , srv946374141:87 , srv177191316:88 , srv1833797085:88 , srv1885150818:88 , srv1897015294:88 , srv2062912588:88 , srv661908597:88 , srv981979757:559 } 2024-04-14 04:53:17,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:17,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv726303018=81, srv1750195808=36, srv1833797085=41, srv1206392108=10, srv1397560139=21, srv2045747975=48, srv716121343=80, srv1446170587=24, srv1885150818=42, srv1755624780=37, srv444143122=67, srv930777275=93, srv989261164=99, srv1412870643=22, srv1267317716=14, srv213212694=52, srv1004982478=0, srv363038237=63, srv826344366=86, srv2125449227=51, srv1034717674=1, srv1897015294=43, srv1045930430=2, srv120233800=8, srv1383432747=20, srv666837819=76, srv1910642058=45, srv365358091=64, srv682453089=77, srv60546987=71, srv820267177=83, srv1717170115=34, srv1381737025=19, srv936715734=94, srv946374141=95, srv661885190=74, srv257256088=55, srv1218889633=11, srv1202808556=9, srv904303876=89, srv351972372=61, srv849573990=87, srv1132968419=5, srv1743540914=35, srv332722509=59, srv699700503=79, srv879968594=88, srv661908597=75, srv128714631=15, srv326781486=58, srv1070687915=3, srv823075226=84, srv928160621=91, srv151240652=26, srv1225898884=12, srv825230235=85, srv235971114=54, srv1643175525=31, srv1189778139=6, srv1287699103=16, srv693142401=78, srv177191316=38, srv643679448=73, srv1123675532=4, srv1950579888=46, srv1802476697=39, srv807633389=82, srv916666736=90, srv354041766=62, srv1626812334=29, srv1201361808=7, srv1376334674=17, srv63346045=72, srv484925100=69, srv2068015187=50, srv981979757=98, srv1669055882=33, srv214885110=53, srv1506852492=25, srv292196539=57, srv402684271=65, srv336140879=60, srv1567673943=28, srv1910300667=44, srv1437517826=23, srv2062912588=49, srv1649494316=32, srv1629823077=30, srv1378973372=18, srv930473294=92, srv1513309329=27, srv483744120=68, srv1808301048=40, srv2011116593=47, srv267492269=56, srv434184021=66, srv947564682=96, srv557588707=70, srv1242303034=13, srv9575988=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-14 04:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-14 04:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-14 04:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-14 04:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-14 04:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-14 04:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-14 04:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-14 04:53:17,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-14 04:53:17,951 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010938046122328536); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:17,951 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-14 04:53:17,951 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.522521582628607E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010938046122328536); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-14 04:53:18,388 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 573 ms to try 5000 different iterations. Found a solution that moves 333 regions; Going from a computed imbalance of 7.522521582628607E-4 to a new imbalance of 4.531158419992326E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:18,389 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1643175525:83 , srv434184021:83 , srv849573990:83 , srv1189778139:84 , srv1206392108:84 , srv128714631:84 , srv1287699103:84 , srv1381737025:84 , srv1383432747:84 , srv1513309329:84 , srv1567673943:84 , srv1629823077:84 , srv1649494316:84 , srv1669055882:84 , srv1755624780:84 , srv177191316:84 , srv1808301048:84 , srv1833797085:84 , srv1885150818:84 , srv1897015294:84 , srv1910642058:84 , srv2062912588:84 , srv213212694:84 , srv235971114:84 , srv257256088:84 , srv326781486:84 , srv336140879:84 , srv351972372:84 , srv444143122:84 , srv60546987:84 , srv661908597:84 , srv699700503:84 , srv726303018:84 , srv823075226:84 , srv825230235:84 , srv928160621:84 , srv930473294:84 , srv936715734:84 , srv946374141:84 , srv947564682:84 , srv1004982478:85 , srv1034717674:85 , srv1045930430:85 , srv1070687915:85 , srv1201361808:85 , srv120233800:85 , srv1218889633:85 , srv1225898884:85 , srv1242303034:85 , srv1267317716:85 , srv1376334674:85 , srv1397560139:85 , srv1412870643:85 , srv1437517826:85 , srv1446170587:85 , srv1506852492:85 , srv1626812334:85 , srv1717170115:85 , srv1743540914:85 , srv1750195808:85 , srv1910300667:85 , srv1950579888:85 , srv2011116593:85 , srv2045747975:85 , srv2068015187:85 , srv2125449227:85 , srv267492269:85 , srv292196539:85 , srv332722509:85 , srv354041766:85 , srv363038237:85 , srv365358091:85 , srv402684271:85 , srv483744120:85 , srv484925100:85 , srv63346045:85 , srv643679448:85 , srv661885190:85 , srv666837819:85 , srv682453089:85 , srv693142401:85 , srv716121343:85 , srv807633389:85 , srv820267177:85 , srv826344366:85 , srv879968594:85 , srv904303876:85 , srv916666736:85 , srv930777275:85 , srv989261164:85 , srv1123675532:86 , srv1132968419:86 , srv1202808556:86 , srv1378973372:86 , srv151240652:86 , srv1802476697:86 , srv214885110:86 , srv557588707:86 , srv9575988:86 , srv981979757:619 } 2024-04-14 04:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:18,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv726303018=81, srv1750195808=36, srv1833797085=41, srv1206392108=10, srv1397560139=21, srv2045747975=48, srv716121343=80, srv1446170587=24, srv1885150818=42, srv1755624780=37, srv444143122=67, srv930777275=93, srv989261164=99, srv1412870643=22, srv1267317716=14, srv213212694=52, srv1004982478=0, srv363038237=63, srv826344366=86, srv2125449227=51, srv1034717674=1, srv1897015294=43, srv1045930430=2, srv120233800=8, srv1383432747=20, srv666837819=76, srv1910642058=45, srv365358091=64, srv682453089=77, srv60546987=71, srv820267177=83, srv1717170115=34, srv1381737025=19, srv936715734=94, srv946374141=95, srv661885190=74, srv257256088=55, srv1218889633=11, srv1202808556=9, srv904303876=89, srv351972372=61, srv849573990=87, srv1132968419=5, srv1743540914=35, srv332722509=59, srv699700503=79, srv879968594=88, srv661908597=75, srv128714631=15, srv326781486=58, srv1070687915=3, srv823075226=84, srv928160621=91, srv151240652=26, srv1225898884=12, srv825230235=85, srv235971114=54, srv1643175525=31, srv1189778139=6, srv1287699103=16, srv693142401=78, srv177191316=38, srv643679448=73, srv1123675532=4, srv1950579888=46, srv1802476697=39, srv807633389=82, srv916666736=90, srv354041766=62, srv1626812334=29, srv1201361808=7, srv1376334674=17, srv63346045=72, srv484925100=69, srv2068015187=50, srv981979757=98, srv1669055882=33, srv214885110=53, srv1506852492=25, srv292196539=57, srv402684271=65, srv336140879=60, srv1567673943=28, srv1910300667=44, srv1437517826=23, srv2062912588=49, srv1649494316=32, srv1629823077=30, srv1378973372=18, srv930473294=92, srv1513309329=27, srv483744120=68, srv1808301048=40, srv2011116593=47, srv267492269=56, srv434184021=66, srv947564682=96, srv557588707=70, srv1242303034=13, srv9575988=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:18,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:18,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:18,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-14 04:53:18,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-14 04:53:18,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-14 04:53:18,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-14 04:53:18,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-14 04:53:18,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-14 04:53:18,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-14 04:53:18,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-14 04:53:18,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-14 04:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-14 04:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-14 04:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-14 04:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-14 04:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-14 04:53:18,539 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:18,540 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-14 04:53:18,540 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.2979868363161375E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-14 04:53:18,762 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 324 ms to try 5000 different iterations. Found a solution that moves 49 regions; Going from a computed imbalance of 4.2979868363161375E-5 to a new imbalance of 7.087174387169283E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0054444444444444445); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:18,763 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2125449227:82 , srv1123675532:83 , srv1132968419:83 , srv2045747975:83 , srv332722509:83 , srv363038237:83 , srv365358091:83 , srv716121343:83 , srv930777275:83 , srv1004982478:84 , srv1034717674:84 , srv1045930430:84 , srv1070687915:84 , srv1189778139:84 , srv1201361808:84 , srv120233800:84 , srv1202808556:84 , srv1218889633:84 , srv1225898884:84 , srv1242303034:84 , srv1267317716:84 , srv128714631:84 , srv1287699103:84 , srv1376334674:84 , srv1378973372:84 , srv1381737025:84 , srv1383432747:84 , srv1397560139:84 , srv1412870643:84 , srv1437517826:84 , srv1446170587:84 , srv1506852492:84 , srv151240652:84 , srv1513309329:84 , srv1567673943:84 , srv1626812334:84 , srv1643175525:84 , srv1669055882:84 , srv1717170115:84 , srv1743540914:84 , srv1750195808:84 , srv1755624780:84 , srv177191316:84 , srv1802476697:84 , srv1808301048:84 , srv1833797085:84 , srv1885150818:84 , srv1897015294:84 , srv1910300667:84 , srv1910642058:84 , srv1950579888:84 , srv2011116593:84 , srv2062912588:84 , srv2068015187:84 , srv213212694:84 , srv214885110:84 , srv267492269:84 , srv292196539:84 , srv351972372:84 , srv354041766:84 , srv402684271:84 , srv444143122:84 , srv483744120:84 , srv557588707:84 , srv60546987:84 , srv63346045:84 , srv643679448:84 , srv661885190:84 , srv661908597:84 , srv666837819:84 , srv682453089:84 , srv693142401:84 , srv699700503:84 , srv726303018:84 , srv807633389:84 , srv820267177:84 , srv823075226:84 , srv825230235:84 , srv826344366:84 , srv879968594:84 , srv904303876:84 , srv916666736:84 , srv928160621:84 , srv930473294:84 , srv936715734:84 , srv946374141:84 , srv947564682:84 , srv9575988:84 , srv989261164:84 , srv1206392108:85 , srv1629823077:85 , srv1649494316:85 , srv235971114:85 , srv257256088:85 , srv326781486:85 , srv336140879:85 , srv434184021:85 , srv484925100:85 , srv849573990:85 , srv981979757:684 } 2024-04-14 04:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv726303018=81, srv1750195808=36, srv1833797085=41, srv1206392108=10, srv1397560139=21, srv2045747975=48, srv716121343=80, srv1446170587=24, srv1885150818=42, srv1755624780=37, srv444143122=67, srv930777275=93, srv989261164=99, srv1412870643=22, srv1267317716=14, srv213212694=52, srv1004982478=0, srv363038237=63, srv826344366=86, srv2125449227=51, srv1034717674=1, srv1897015294=43, srv1045930430=2, srv120233800=8, srv1383432747=20, srv666837819=76, srv1910642058=45, srv365358091=64, srv682453089=77, srv60546987=71, srv820267177=83, srv1717170115=34, srv1381737025=19, srv936715734=94, srv946374141=95, srv661885190=74, srv257256088=55, srv1218889633=11, srv1202808556=9, srv904303876=89, srv351972372=61, srv849573990=87, srv1132968419=5, srv1743540914=35, srv332722509=59, srv699700503=79, srv879968594=88, srv661908597=75, srv128714631=15, srv326781486=58, srv1070687915=3, srv823075226=84, srv928160621=91, srv151240652=26, srv1225898884=12, srv825230235=85, srv235971114=54, srv1643175525=31, srv1189778139=6, srv1287699103=16, srv693142401=78, srv177191316=38, srv643679448=73, srv1123675532=4, srv1950579888=46, srv1802476697=39, srv807633389=82, srv916666736=90, srv354041766=62, srv1626812334=29, srv1201361808=7, srv1376334674=17, srv63346045=72, srv484925100=69, srv2068015187=50, srv981979757=98, srv1669055882=33, srv214885110=53, srv1506852492=25, srv292196539=57, srv402684271=65, srv336140879=60, srv1567673943=28, srv1910300667=44, srv1437517826=23, srv2062912588=49, srv1649494316=32, srv1629823077=30, srv1378973372=18, srv930473294=92, srv1513309329=27, srv483744120=68, srv1808301048=40, srv2011116593=47, srv267492269=56, srv434184021=66, srv947564682=96, srv557588707=70, srv1242303034=13, srv9575988=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:18,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:18,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-14 04:53:18,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-14 04:53:18,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-14 04:53:18,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-14 04:53:18,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-14 04:53:18,888 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:18,888 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2125449227:82 , srv1123675532:83 , srv1132968419:83 , srv2045747975:83 , srv332722509:83 , srv363038237:83 , srv365358091:83 , srv716121343:83 , srv930777275:83 , srv1004982478:84 , srv1034717674:84 , srv1045930430:84 , srv1070687915:84 , srv1189778139:84 , srv1201361808:84 , srv120233800:84 , srv1202808556:84 , srv1218889633:84 , srv1225898884:84 , srv1242303034:84 , srv1267317716:84 , srv128714631:84 , srv1287699103:84 , srv1376334674:84 , srv1378973372:84 , srv1381737025:84 , srv1383432747:84 , srv1397560139:84 , srv1412870643:84 , srv1437517826:84 , srv1446170587:84 , srv1506852492:84 , srv151240652:84 , srv1513309329:84 , srv1567673943:84 , srv1626812334:84 , srv1643175525:84 , srv1669055882:84 , srv1717170115:84 , srv1743540914:84 , srv1750195808:84 , srv1755624780:84 , srv177191316:84 , srv1802476697:84 , srv1808301048:84 , srv1833797085:84 , srv1885150818:84 , srv1897015294:84 , srv1910300667:84 , srv1910642058:84 , srv1950579888:84 , srv2011116593:84 , srv2062912588:84 , srv2068015187:84 , srv213212694:84 , srv214885110:84 , srv267492269:84 , srv292196539:84 , srv351972372:84 , srv354041766:84 , srv402684271:84 , srv444143122:84 , srv483744120:84 , srv557588707:84 , srv60546987:84 , srv63346045:84 , srv643679448:84 , srv661885190:84 , srv661908597:84 , srv666837819:84 , srv682453089:84 , srv693142401:84 , srv699700503:84 , srv726303018:84 , srv807633389:84 , srv820267177:84 , srv823075226:84 , srv825230235:84 , srv826344366:84 , srv879968594:84 , srv904303876:84 , srv916666736:84 , srv928160621:84 , srv930473294:84 , srv936715734:84 , srv946374141:84 , srv947564682:84 , srv9575988:84 , srv989261164:84 , srv1206392108:85 , srv1629823077:85 , srv1649494316:85 , srv235971114:85 , srv257256088:85 , srv326781486:85 , srv336140879:85 , srv434184021:85 , srv484925100:85 , srv849573990:85 , srv981979757:684 } 2024-04-14 04:53:18,913 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=341 (was 319) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7859 (was 8644) 2024-04-14 04:53:18,922 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=341, ProcessCount=169, AvailableMemoryMB=7859 2024-04-14 04:53:18,923 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-14 04:53:18,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-14 04:53:18,934 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1025339901:84 , srv1051162757:84 , srv1112742829:84 , srv1181270676:84 , srv1309463997:84 , srv1361191267:84 , srv1375567442:84 , srv139504947:84 , srv1472004764:84 , srv1491872353:84 , srv1513268377:84 , srv1631294329:84 , srv1734963451:84 , srv1763744207:84 , srv2002154188:84 , srv2098627487:84 , srv2114828426:84 , srv268098497:84 , srv269550083:84 , srv270949201:84 , srv438236827:84 , srv504321159:84 , srv540586300:84 , srv588119666:84 , srv615060158:84 , srv645146150:84 , srv671704413:84 , srv683723877:84 , srv750057780:84 , srv1573119905:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-14 04:53:18,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:53:18,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv504321159=22, srv1025339901=0, srv1734963451=13, srv671704413=27, srv139504947=7, srv2002154188=15, srv615060158=25, srv2114828426=17, srv1112742829=2, srv540586300=23, srv270949201=20, srv1375567442=6, srv438236827=21, srv1631294329=12, srv1051162757=1, srv268098497=18, srv1472004764=8, srv2098627487=16, srv1513268377=10, srv1361191267=5, srv269550083=19, srv588119666=24, srv1491872353=9, srv1763744207=14, srv645146150=26, srv750057780=29, srv1309463997=4, srv1181270676=3, srv1573119905=11, srv683723877=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:53:18,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:53:18,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:53:18,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:53:18,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-14 04:53:18,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.078096127015611, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:53:18,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912439121610434, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.078096127015611, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-14 04:55:18,955 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6236822 different iterations. Found a solution that moves 1861 regions; Going from a computed imbalance of 0.9912439121610434 to a new imbalance of 4.3436668390529226E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6892592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:55:18,963 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1025339901:90 , srv1051162757:90 , srv1112742829:90 , srv1181270676:90 , srv1309463997:90 , srv1361191267:90 , srv1375567442:90 , srv139504947:90 , srv1472004764:90 , srv1491872353:90 , srv1513268377:90 , srv1573119905:90 , srv1631294329:90 , srv1734963451:90 , srv1763744207:90 , srv2002154188:90 , srv2098627487:90 , srv2114828426:90 , srv268098497:90 , srv269550083:90 , srv270949201:90 , srv438236827:90 , srv504321159:90 , srv540586300:90 , srv588119666:90 , srv615060158:90 , srv645146150:90 , srv671704413:90 , srv683723877:90 , srv750057780:90 } 2024-04-14 04:55:18,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-14 04:55:18,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv504321159=22, srv1025339901=0, srv1734963451=13, srv671704413=27, srv139504947=7, srv2002154188=15, srv615060158=25, srv2114828426=17, srv1112742829=2, srv540586300=23, srv270949201=20, srv1375567442=6, srv438236827=21, srv1631294329=12, srv1051162757=1, srv268098497=18, srv1472004764=8, srv2098627487=16, srv1513268377=10, srv1361191267=5, srv269550083=19, srv588119666=24, srv1491872353=9, srv1763744207=14, srv645146150=26, srv750057780=29, srv1309463997=4, srv1181270676=3, srv1573119905=11, srv683723877=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-14 04:55:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-14 04:55:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-14 04:55:18,980 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-14 04:55:18,980 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1025339901:90 , srv1051162757:90 , srv1112742829:90 , srv1181270676:90 , srv1309463997:90 , srv1361191267:90 , srv1375567442:90 , srv139504947:90 , srv1472004764:90 , srv1491872353:90 , srv1513268377:90 , srv1573119905:90 , srv1631294329:90 , srv1734963451:90 , srv1763744207:90 , srv2002154188:90 , srv2098627487:90 , srv2114828426:90 , srv268098497:90 , srv269550083:90 , srv270949201:90 , srv438236827:90 , srv504321159:90 , srv540586300:90 , srv588119666:90 , srv615060158:90 , srv645146150:90 , srv671704413:90 , srv683723877:90 , srv750057780:90 } 2024-04-14 04:55:18,996 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 341), ProcessCount=167 (was 169), AvailableMemoryMB=7169 (was 7859)