2024-01-09 19:53:16,366 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-09 19:53:16,693 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-09 19:53:16,813 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-09 19:53:16,876 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-09 19:53:16,884 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-09 19:53:16,897 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=259, ProcessCount=167, AvailableMemoryMB=10156 2024-01-09 19:53:16,899 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-09 19:53:16,899 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-09 19:53:17,042 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009316774:84 , srv1009797333:84 , srv1018419554:84 , srv1030703165:84 , srv1043996351:84 , srv1048524665:84 , srv1065159899:84 , srv1076704627:84 , srv108812887:84 , srv111146526:84 , srv1139578995:84 , srv1151002670:84 , srv1193131892:84 , srv1243328901:84 , srv1257443000:84 , srv1264014893:84 , srv1278620546:84 , srv1279072700:84 , srv1299455103:84 , srv133627406:84 , srv1340861712:84 , srv1353355129:84 , srv136580878:84 , srv1413172720:84 , srv142077432:84 , srv1456809519:84 , srv1465636586:84 , srv1470301453:84 , srv1470639540:84 , srv1541571223:84 , srv1548430260:84 , srv1551284048:84 , srv1556046759:84 , srv1564435385:84 , srv1577939912:84 , srv1582215061:84 , srv1586947150:84 , srv1593182298:84 , srv16316557:84 , srv1668702937:84 , srv1693535662:84 , srv1755410198:84 , srv1764306136:84 , srv1796918536:84 , srv1813070981:84 , srv183122115:84 , srv1838142121:84 , srv1860054335:84 , srv1918940434:84 , srv1939462061:84 , srv1972053186:84 , srv1986460637:84 , srv1995542006:84 , srv2018019381:84 , srv2020478240:84 , srv2030171949:84 , srv2040443846:84 , srv2049345441:84 , srv2086999720:84 , srv2096517867:84 , srv2103959201:84 , srv2129149557:84 , srv2133703401:84 , srv228151844:84 , srv246022539:84 , srv273440243:84 , srv308875350:84 , srv340154366:84 , srv367505300:84 , srv437897660:84 , srv440250745:84 , srv44500183:84 , srv447934186:84 , srv455504907:84 , srv46619387:84 , srv506519243:84 , srv567897813:84 , srv599929518:84 , srv601882767:84 , srv612812569:84 , srv628801870:84 , srv661199473:84 , srv746201953:84 , srv751516154:84 , srv752689104:84 , srv777826649:84 , srv778093102:84 , srv788911488:84 , srv79020620:84 , srv803818603:84 , srv807254032:84 , srv817645492:84 , srv845627175:84 , srv848622314:84 , srv85923193:84 , srv88068495:84 , srv943165002:84 , srv943497899:84 , srv981809533:84 , srv1236757847:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-09 19:53:17,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:53:17,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1340861712=21, srv1986460637=52, srv845627175=93, srv1030703165=3, srv848622314=94, srv2086999720=59, srv567897813=77, srv1048524665=5, srv367505300=69, srv1796918536=44, srv2103959201=61, srv1918940434=49, srv16316557=39, srv1541571223=30, srv1353355129=22, srv1755410198=42, srv506519243=76, srv1813070981=45, srv1564435385=34, srv183122115=46, srv599929518=78, srv1577939912=35, srv803818603=90, srv308875350=67, srv1279072700=18, srv2040443846=57, srv228151844=64, srv79020620=89, srv788911488=88, srv1548430260=31, srv777826649=86, srv1838142121=47, srv1076704627=7, srv437897660=70, srv817645492=92, srv1236757847=13, srv1009797333=1, srv612812569=80, srv943497899=98, srv1065159899=6, srv1413172720=24, srv1264014893=16, srv447934186=73, srv2133703401=63, srv1243328901=14, srv111146526=9, srv2129149557=62, srv1995542006=53, srv88068495=96, srv1257443000=15, srv440250745=71, srv1139578995=10, srv1693535662=41, srv133627406=20, srv628801870=81, srv340154366=68, srv981809533=99, srv1465636586=27, srv1593182298=38, srv1193131892=12, srv1456809519=26, srv661199473=82, srv746201953=83, srv1972053186=51, srv1043996351=4, srv2096517867=60, srv1018419554=2, srv1668702937=40, srv273440243=66, srv2030171949=56, srv1470639540=29, srv2020478240=55, srv778093102=87, srv1586947150=37, srv1860054335=48, srv85923193=95, srv752689104=85, srv2018019381=54, srv1556046759=33, srv142077432=25, srv1151002670=11, srv1009316774=0, srv601882767=79, srv1582215061=36, srv1939462061=50, srv246022539=65, srv44500183=72, srv943165002=97, srv108812887=8, srv136580878=23, srv1299455103=19, srv751516154=84, srv46619387=75, srv1278620546=17, srv1470301453=28, srv1551284048=32, srv1764306136=43, srv2049345441=58, srv807254032=91, srv455504907=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:53:17,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:53:17,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:53:17,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:53:17,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:53:17,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:53:17,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:53:17,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:53:17,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:53:17,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:53:17,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:53:17,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:53:17,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:53:17,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:53:17,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:53:17,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:53:17,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:53:17,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:53:17,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:53:17,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 19:53:17,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 19:53:17,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 19:53:17,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 19:53:17,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 19:53:17,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 19:53:17,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 19:53:17,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 19:53:17,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 19:53:17,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 19:53:17,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 19:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 19:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 19:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 19:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 19:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 19:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 19:53:17,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 19:53:17,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 19:53:17,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 19:53:17,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 19:53:17,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 19:53:17,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 19:53:17,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 19:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 19:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 19:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 19:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 19:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 19:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 19:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 19:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 19:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 19:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 19:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 19:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 19:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 19:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 19:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 19:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 19:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 19:53:17,419 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:17,420 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 19:53:17,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 19:53:17,875 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 676 ms to try 5000 different iterations. Found a solution that moves 4669 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17943842290994277. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04106755966593129, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058424892070657805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5187777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04106755966593129, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24766666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:17,905 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1548430260:82 , srv1838142121:82 , srv273440243:82 , srv506519243:82 , srv1139578995:83 , srv1668702937:83 , srv1860054335:83 , srv1995542006:83 , srv2018019381:83 , srv2040443846:83 , srv447934186:83 , srv455504907:83 , srv788911488:83 , srv848622314:83 , srv981809533:83 , srv1076704627:84 , srv136580878:84 , srv1470639540:84 , srv1541571223:84 , srv16316557:84 , srv1755410198:84 , srv1796918536:84 , srv183122115:84 , srv661199473:84 , srv751516154:84 , srv1030703165:85 , srv1043996351:85 , srv1048524665:85 , srv111146526:85 , srv1264014893:85 , srv133627406:85 , srv1470301453:85 , srv1551284048:85 , srv1586947150:85 , srv1593182298:85 , srv1939462061:85 , srv2030171949:85 , srv340154366:85 , srv440250745:85 , srv44500183:85 , srv777826649:85 , srv803818603:85 , srv817645492:85 , srv943165002:85 , srv1018419554:86 , srv1065159899:86 , srv1243328901:86 , srv1257443000:86 , srv1465636586:86 , srv1582215061:86 , srv1764306136:86 , srv1972053186:86 , srv228151844:86 , srv612812569:86 , srv778093102:86 , srv1009316774:87 , srv1193131892:87 , srv1278620546:87 , srv1279072700:87 , srv1299455103:87 , srv1456809519:87 , srv1693535662:87 , srv1813070981:87 , srv2020478240:87 , srv2096517867:87 , srv2103959201:87 , srv2129149557:87 , srv437897660:87 , srv85923193:87 , srv108812887:88 , srv1340861712:88 , srv1564435385:88 , srv1577939912:88 , srv2049345441:88 , srv2086999720:88 , srv367505300:88 , srv746201953:88 , srv752689104:88 , srv79020620:88 , srv845627175:88 , srv943497899:88 , srv1353355129:89 , srv1413172720:89 , srv2133703401:89 , srv308875350:89 , srv601882767:89 , srv88068495:89 , srv142077432:90 , srv567897813:90 , srv599929518:90 , srv628801870:90 , srv1986460637:91 , srv246022539:91 , srv46619387:91 , srv1151002670:92 , srv1556046759:92 , srv807254032:92 , srv1009797333:93 , srv1918940434:93 , srv1236757847:455 } 2024-01-09 19:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:53:17,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1340861712=21, srv1986460637=52, srv845627175=93, srv1030703165=3, srv848622314=94, srv2086999720=59, srv567897813=77, srv1048524665=5, srv367505300=69, srv1796918536=44, srv2103959201=61, srv1918940434=49, srv16316557=39, srv1541571223=30, srv1353355129=22, srv1755410198=42, srv506519243=76, srv1813070981=45, srv1564435385=34, srv183122115=46, srv599929518=78, srv1577939912=35, srv803818603=90, srv308875350=67, srv1279072700=18, srv2040443846=57, srv228151844=64, srv79020620=89, srv788911488=88, srv1548430260=31, srv777826649=86, srv1838142121=47, srv1076704627=7, srv437897660=70, srv817645492=92, srv1236757847=13, srv1009797333=1, srv612812569=80, srv943497899=98, srv1065159899=6, srv1413172720=24, srv1264014893=16, srv447934186=73, srv2133703401=63, srv1243328901=14, srv111146526=9, srv2129149557=62, srv1995542006=53, srv88068495=96, srv1257443000=15, srv440250745=71, srv1139578995=10, srv1693535662=41, srv133627406=20, srv628801870=81, srv340154366=68, srv981809533=99, srv1465636586=27, srv1593182298=38, srv1193131892=12, srv1456809519=26, srv661199473=82, srv746201953=83, srv1972053186=51, srv1043996351=4, srv2096517867=60, srv1018419554=2, srv1668702937=40, srv273440243=66, srv2030171949=56, srv1470639540=29, srv2020478240=55, srv778093102=87, srv1586947150=37, srv1860054335=48, srv85923193=95, srv752689104=85, srv2018019381=54, srv1556046759=33, srv142077432=25, srv1151002670=11, srv1009316774=0, srv601882767=79, srv1582215061=36, srv1939462061=50, srv246022539=65, srv44500183=72, srv943165002=97, srv108812887=8, srv136580878=23, srv1299455103=19, srv751516154=84, srv46619387=75, srv1278620546=17, srv1470301453=28, srv1551284048=32, srv1764306136=43, srv2049345441=58, srv807254032=91, srv455504907=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:53:17,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 19:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 19:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 19:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 19:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 19:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 19:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 19:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 19:53:18,194 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04106755966593129, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058424892070657805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04106755966593129, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24766666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:18,195 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 19:53:18,195 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17940572987318051, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04106755966593129, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058424892070657805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04106755966593129, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24766666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 19:53:18,462 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 486 ms to try 5000 different iterations. Found a solution that moves 2633 regions; Going from a computed imbalance of 0.17940572987318051 to a new imbalance of 0.015746914528311916. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020597146021777486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03210532610152554, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29255555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020597146021777486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:18,475 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv807254032:78 , srv1556046759:79 , srv1918940434:79 , srv1151002670:80 , srv1986460637:80 , srv567897813:81 , srv1009797333:82 , srv246022539:82 , srv308875350:82 , srv46619387:82 , srv628801870:82 , srv108812887:83 , srv1813070981:83 , srv2086999720:83 , srv2133703401:83 , srv599929518:83 , srv845627175:83 , srv88068495:83 , srv1193131892:84 , srv1257443000:84 , srv1278620546:84 , srv1299455103:84 , srv1340861712:84 , srv1353355129:84 , srv142077432:84 , srv1456809519:84 , srv1564435385:84 , srv1764306136:84 , srv2103959201:84 , srv2129149557:84 , srv752689104:84 , srv79020620:84 , srv1009316774:85 , srv133627406:85 , srv1413172720:85 , srv1693535662:85 , srv2049345441:85 , srv2096517867:85 , srv367505300:85 , srv437897660:85 , srv612812569:85 , srv746201953:85 , srv803818603:85 , srv85923193:85 , srv943497899:85 , srv1043996351:86 , srv1065159899:86 , srv1279072700:86 , srv1465636586:86 , srv1582215061:86 , srv1586947150:86 , srv228151844:86 , srv601882767:86 , srv817645492:86 , srv943165002:86 , srv1018419554:87 , srv1076704627:87 , srv111146526:87 , srv1243328901:87 , srv136580878:87 , srv1470301453:87 , srv1470639540:87 , srv1668702937:87 , srv1755410198:87 , srv183122115:87 , srv1972053186:87 , srv2020478240:87 , srv2030171949:87 , srv661199473:87 , srv777826649:87 , srv778093102:87 , srv1030703165:88 , srv1048524665:88 , srv1541571223:88 , srv1551284048:88 , srv1577939912:88 , srv1593182298:88 , srv2018019381:88 , srv340154366:88 , srv44500183:88 , srv447934186:88 , srv455504907:88 , srv788911488:88 , srv1139578995:89 , srv1264014893:89 , srv16316557:89 , srv1796918536:89 , srv1995542006:89 , srv273440243:89 , srv440250745:89 , srv848622314:89 , srv1548430260:90 , srv1838142121:90 , srv1939462061:90 , srv751516154:90 , srv1860054335:91 , srv506519243:91 , srv2040443846:92 , srv981809533:95 , srv1236757847:502 } 2024-01-09 19:53:18,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:53:18,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1340861712=21, srv1986460637=52, srv845627175=93, srv1030703165=3, srv848622314=94, srv2086999720=59, srv567897813=77, srv1048524665=5, srv367505300=69, srv1796918536=44, srv2103959201=61, srv1918940434=49, srv16316557=39, srv1541571223=30, srv1353355129=22, srv1755410198=42, srv506519243=76, srv1813070981=45, srv1564435385=34, srv183122115=46, srv599929518=78, srv1577939912=35, srv803818603=90, srv308875350=67, srv1279072700=18, srv2040443846=57, srv228151844=64, srv79020620=89, srv788911488=88, srv1548430260=31, srv777826649=86, srv1838142121=47, srv1076704627=7, srv437897660=70, srv817645492=92, srv1236757847=13, srv1009797333=1, srv612812569=80, srv943497899=98, srv1065159899=6, srv1413172720=24, srv1264014893=16, srv447934186=73, srv2133703401=63, srv1243328901=14, srv111146526=9, srv2129149557=62, srv1995542006=53, srv88068495=96, srv1257443000=15, srv440250745=71, srv1139578995=10, srv1693535662=41, srv133627406=20, srv628801870=81, srv340154366=68, srv981809533=99, srv1465636586=27, srv1593182298=38, srv1193131892=12, srv1456809519=26, srv661199473=82, srv746201953=83, srv1972053186=51, srv1043996351=4, srv2096517867=60, srv1018419554=2, srv1668702937=40, srv273440243=66, srv2030171949=56, srv1470639540=29, srv2020478240=55, srv778093102=87, srv1586947150=37, srv1860054335=48, srv85923193=95, srv752689104=85, srv2018019381=54, srv1556046759=33, srv142077432=25, srv1151002670=11, srv1009316774=0, srv601882767=79, srv1582215061=36, srv1939462061=50, srv246022539=65, srv44500183=72, srv943165002=97, srv108812887=8, srv136580878=23, srv1299455103=19, srv751516154=84, srv46619387=75, srv1278620546=17, srv1470301453=28, srv1551284048=32, srv1764306136=43, srv2049345441=58, srv807254032=91, srv455504907=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:53:18,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:53:18,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:53:18,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:53:18,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:53:18,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:53:18,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 19:53:18,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 19:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 19:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 19:53:18,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 19:53:18,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 19:53:18,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 19:53:18,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 19:53:18,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 19:53:18,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 19:53:18,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 19:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 19:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 19:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 19:53:18,671 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020597146021777486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03210532610152554, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020597146021777486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:18,671 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 19:53:18,671 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01572847786825728, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020597146021777486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03210532610152554, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020597146021777486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 19:53:19,024 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 487 ms to try 5000 different iterations. Found a solution that moves 1000 regions; Going from a computed imbalance of 0.01572847786825728 to a new imbalance of 8.771306957911359E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005658970147920401); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008914105645620285); MoveCostFunction : (multiplier=7.0, imbalance=0.1111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005658970147920401); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:19,026 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv981809533:80 , srv1009797333:82 , srv2040443846:82 , srv142077432:83 , srv1577939912:83 , srv1860054335:83 , srv1939462061:83 , srv367505300:83 , srv440250745:83 , srv601882767:83 , srv751516154:83 , srv1030703165:84 , srv1048524665:84 , srv1243328901:84 , srv1264014893:84 , srv1551284048:84 , srv2020478240:84 , srv340154366:84 , srv46619387:84 , srv79020620:84 , srv1009316774:85 , srv1018419554:85 , srv1065159899:85 , srv111146526:85 , srv1139578995:85 , srv1279072700:85 , srv1340861712:85 , srv1353355129:85 , srv1413172720:85 , srv1470301453:85 , srv1541571223:85 , srv1548430260:85 , srv1564435385:85 , srv1582215061:85 , srv1593182298:85 , srv16316557:85 , srv1693535662:85 , srv1796918536:85 , srv1918940434:85 , srv1972053186:85 , srv1995542006:85 , srv2030171949:85 , srv2049345441:85 , srv2086999720:85 , srv2133703401:85 , srv246022539:85 , srv44500183:85 , srv447934186:85 , srv506519243:85 , srv599929518:85 , srv746201953:85 , srv777826649:85 , srv778093102:85 , srv848622314:85 , srv85923193:85 , srv88068495:85 , srv943497899:85 , srv1076704627:86 , srv1151002670:86 , srv1278620546:86 , srv1299455103:86 , srv136580878:86 , srv1465636586:86 , srv1556046759:86 , srv1586947150:86 , srv1986460637:86 , srv2096517867:86 , srv228151844:86 , srv273440243:86 , srv437897660:86 , srv612812569:86 , srv628801870:86 , srv752689104:86 , srv788911488:86 , srv845627175:86 , srv1043996351:87 , srv108812887:87 , srv1193131892:87 , srv133627406:87 , srv1470639540:87 , srv1668702937:87 , srv1755410198:87 , srv1813070981:87 , srv183122115:87 , srv1838142121:87 , srv2018019381:87 , srv2103959201:87 , srv2129149557:87 , srv308875350:87 , srv455504907:87 , srv567897813:87 , srv661199473:87 , srv803818603:87 , srv807254032:87 , srv817645492:87 , srv943165002:87 , srv1257443000:88 , srv1456809519:88 , srv1764306136:89 , srv1236757847:551 } 2024-01-09 19:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1340861712=21, srv1986460637=52, srv845627175=93, srv1030703165=3, srv848622314=94, srv2086999720=59, srv567897813=77, srv1048524665=5, srv367505300=69, srv1796918536=44, srv2103959201=61, srv1918940434=49, srv16316557=39, srv1541571223=30, srv1353355129=22, srv1755410198=42, srv506519243=76, srv1813070981=45, srv1564435385=34, srv183122115=46, srv599929518=78, srv1577939912=35, srv803818603=90, srv308875350=67, srv1279072700=18, srv2040443846=57, srv228151844=64, srv79020620=89, srv788911488=88, srv1548430260=31, srv777826649=86, srv1838142121=47, srv1076704627=7, srv437897660=70, srv817645492=92, srv1236757847=13, srv1009797333=1, srv612812569=80, srv943497899=98, srv1065159899=6, srv1413172720=24, srv1264014893=16, srv447934186=73, srv2133703401=63, srv1243328901=14, srv111146526=9, srv2129149557=62, srv1995542006=53, srv88068495=96, srv1257443000=15, srv440250745=71, srv1139578995=10, srv1693535662=41, srv133627406=20, srv628801870=81, srv340154366=68, srv981809533=99, srv1465636586=27, srv1593182298=38, srv1193131892=12, srv1456809519=26, srv661199473=82, srv746201953=83, srv1972053186=51, srv1043996351=4, srv2096517867=60, srv1018419554=2, srv1668702937=40, srv273440243=66, srv2030171949=56, srv1470639540=29, srv2020478240=55, srv778093102=87, srv1586947150=37, srv1860054335=48, srv85923193=95, srv752689104=85, srv2018019381=54, srv1556046759=33, srv142077432=25, srv1151002670=11, srv1009316774=0, srv601882767=79, srv1582215061=36, srv1939462061=50, srv246022539=65, srv44500183=72, srv943165002=97, srv108812887=8, srv136580878=23, srv1299455103=19, srv751516154=84, srv46619387=75, srv1278620546=17, srv1470301453=28, srv1551284048=32, srv1764306136=43, srv2049345441=58, srv807254032=91, srv455504907=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:53:19,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:53:19,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 19:53:19,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 19:53:19,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 19:53:19,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 19:53:19,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 19:53:19,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 19:53:19,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 19:53:19,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 19:53:19,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 19:53:19,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 19:53:19,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 19:53:19,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 19:53:19,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 19:53:19,156 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005658970147920401); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008914105645620285); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005658970147920401); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:19,156 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-09 19:53:19,157 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.701285461311903E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005658970147920401); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008914105645620285); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005658970147920401); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-09 19:53:19,396 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 213 regions; Going from a computed imbalance of 8.701285461311903E-4 to a new imbalance of 2.1454117104809748E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.023666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:19,397 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1413172720:83 , srv1456809519:83 , srv1577939912:83 , srv1764306136:83 , srv2103959201:83 , srv599929518:83 , srv628801870:83 , srv1018419554:84 , srv1043996351:84 , srv108812887:84 , srv1151002670:84 , srv1257443000:84 , srv1264014893:84 , srv1279072700:84 , srv1353355129:84 , srv1465636586:84 , srv1470639540:84 , srv1582215061:84 , srv1593182298:84 , srv16316557:84 , srv1755410198:84 , srv1796918536:84 , srv183122115:84 , srv1838142121:84 , srv1939462061:84 , srv1972053186:84 , srv1995542006:84 , srv2018019381:84 , srv2020478240:84 , srv2049345441:84 , srv2096517867:84 , srv2129149557:84 , srv228151844:84 , srv246022539:84 , srv308875350:84 , srv44500183:84 , srv455504907:84 , srv506519243:84 , srv567897813:84 , srv601882767:84 , srv661199473:84 , srv746201953:84 , srv752689104:84 , srv778093102:84 , srv807254032:84 , srv817645492:84 , srv848622314:84 , srv943165002:84 , srv943497899:84 , srv1009316774:85 , srv1009797333:85 , srv1030703165:85 , srv1048524665:85 , srv1076704627:85 , srv111146526:85 , srv1139578995:85 , srv1193131892:85 , srv1243328901:85 , srv1278620546:85 , srv1299455103:85 , srv133627406:85 , srv1340861712:85 , srv136580878:85 , srv142077432:85 , srv1470301453:85 , srv1541571223:85 , srv1548430260:85 , srv1551284048:85 , srv1556046759:85 , srv1564435385:85 , srv1586947150:85 , srv1668702937:85 , srv1693535662:85 , srv1813070981:85 , srv1860054335:85 , srv1918940434:85 , srv1986460637:85 , srv2030171949:85 , srv2040443846:85 , srv2133703401:85 , srv340154366:85 , srv437897660:85 , srv440250745:85 , srv46619387:85 , srv751516154:85 , srv777826649:85 , srv788911488:85 , srv803818603:85 , srv845627175:85 , srv85923193:85 , srv88068495:85 , srv981809533:85 , srv1065159899:86 , srv2086999720:86 , srv273440243:86 , srv367505300:86 , srv447934186:86 , srv612812569:86 , srv79020620:86 , srv1236757847:634 } 2024-01-09 19:53:19,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:53:19,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1340861712=21, srv1986460637=52, srv845627175=93, srv1030703165=3, srv848622314=94, srv2086999720=59, srv567897813=77, srv1048524665=5, srv367505300=69, srv1796918536=44, srv2103959201=61, srv1918940434=49, srv16316557=39, srv1541571223=30, srv1353355129=22, srv1755410198=42, srv506519243=76, srv1813070981=45, srv1564435385=34, srv183122115=46, srv599929518=78, srv1577939912=35, srv803818603=90, srv308875350=67, srv1279072700=18, srv2040443846=57, srv228151844=64, srv79020620=89, srv788911488=88, srv1548430260=31, srv777826649=86, srv1838142121=47, srv1076704627=7, srv437897660=70, srv817645492=92, srv1236757847=13, srv1009797333=1, srv612812569=80, srv943497899=98, srv1065159899=6, srv1413172720=24, srv1264014893=16, srv447934186=73, srv2133703401=63, srv1243328901=14, srv111146526=9, srv2129149557=62, srv1995542006=53, srv88068495=96, srv1257443000=15, srv440250745=71, srv1139578995=10, srv1693535662=41, srv133627406=20, srv628801870=81, srv340154366=68, srv981809533=99, srv1465636586=27, srv1593182298=38, srv1193131892=12, srv1456809519=26, srv661199473=82, srv746201953=83, srv1972053186=51, srv1043996351=4, srv2096517867=60, srv1018419554=2, srv1668702937=40, srv273440243=66, srv2030171949=56, srv1470639540=29, srv2020478240=55, srv778093102=87, srv1586947150=37, srv1860054335=48, srv85923193=95, srv752689104=85, srv2018019381=54, srv1556046759=33, srv142077432=25, srv1151002670=11, srv1009316774=0, srv601882767=79, srv1582215061=36, srv1939462061=50, srv246022539=65, srv44500183=72, srv943165002=97, srv108812887=8, srv136580878=23, srv1299455103=19, srv751516154=84, srv46619387=75, srv1278620546=17, srv1470301453=28, srv1551284048=32, srv1764306136=43, srv2049345441=58, srv807254032=91, srv455504907=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:53:19,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:53:19,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:53:19,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:53:19,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:53:19,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:53:19,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-09 19:53:19,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-09 19:53:19,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-09 19:53:19,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-09 19:53:19,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-09 19:53:19,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-09 19:53:19,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-09 19:53:19,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-09 19:53:19,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-09 19:53:19,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-09 19:53:19,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-09 19:53:19,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-09 19:53:19,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-09 19:53:19,528 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9962659227241334E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:19,529 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1413172720:83 , srv1456809519:83 , srv1577939912:83 , srv1764306136:83 , srv2103959201:83 , srv599929518:83 , srv628801870:83 , srv1018419554:84 , srv1043996351:84 , srv108812887:84 , srv1151002670:84 , srv1257443000:84 , srv1264014893:84 , srv1279072700:84 , srv1353355129:84 , srv1465636586:84 , srv1470639540:84 , srv1582215061:84 , srv1593182298:84 , srv16316557:84 , srv1755410198:84 , srv1796918536:84 , srv183122115:84 , srv1838142121:84 , srv1939462061:84 , srv1972053186:84 , srv1995542006:84 , srv2018019381:84 , srv2020478240:84 , srv2049345441:84 , srv2096517867:84 , srv2129149557:84 , srv228151844:84 , srv246022539:84 , srv308875350:84 , srv44500183:84 , srv455504907:84 , srv506519243:84 , srv567897813:84 , srv601882767:84 , srv661199473:84 , srv746201953:84 , srv752689104:84 , srv778093102:84 , srv807254032:84 , srv817645492:84 , srv848622314:84 , srv943165002:84 , srv943497899:84 , srv1009316774:85 , srv1009797333:85 , srv1030703165:85 , srv1048524665:85 , srv1076704627:85 , srv111146526:85 , srv1139578995:85 , srv1193131892:85 , srv1243328901:85 , srv1278620546:85 , srv1299455103:85 , srv133627406:85 , srv1340861712:85 , srv136580878:85 , srv142077432:85 , srv1470301453:85 , srv1541571223:85 , srv1548430260:85 , srv1551284048:85 , srv1556046759:85 , srv1564435385:85 , srv1586947150:85 , srv1668702937:85 , srv1693535662:85 , srv1813070981:85 , srv1860054335:85 , srv1918940434:85 , srv1986460637:85 , srv2030171949:85 , srv2040443846:85 , srv2133703401:85 , srv340154366:85 , srv437897660:85 , srv440250745:85 , srv46619387:85 , srv751516154:85 , srv777826649:85 , srv788911488:85 , srv803818603:85 , srv845627175:85 , srv85923193:85 , srv88068495:85 , srv981809533:85 , srv1065159899:86 , srv2086999720:86 , srv273440243:86 , srv367505300:86 , srv447934186:86 , srv612812569:86 , srv79020620:86 , srv1236757847:634 } 2024-01-09 19:53:19,556 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=259 (was 259), ProcessCount=167 (was 167), AvailableMemoryMB=9549 (was 10156) 2024-01-09 19:53:19,564 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=259, ProcessCount=167, AvailableMemoryMB=9548 2024-01-09 19:53:19,565 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-09 19:53:19,565 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-09 19:53:19,572 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1061502750:84 , srv1078512979:84 , srv1205086099:84 , srv1350499850:84 , srv1415135134:84 , srv1417162432:84 , srv1427411420:84 , srv1483419278:84 , srv1563172923:84 , srv1582021464:84 , srv17392640:84 , srv1833622351:84 , srv184755129:84 , srv1861189694:84 , srv1993005034:84 , srv2052841128:84 , srv208000225:84 , srv237831772:84 , srv281256306:84 , srv465487133:84 , srv508181454:84 , srv537478003:84 , srv545318246:84 , srv661785619:84 , srv664507623:84 , srv74160577:84 , srv782157877:84 , srv798693963:84 , srv926741660:84 , srv1430332521:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-09 19:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1078512979=1, srv1415135134=4, srv661785619=24, srv508181454=21, srv17392640=11, srv1061502750=0, srv1430332521=7, srv1417162432=5, srv2052841128=16, srv74160577=26, srv537478003=22, srv281256306=19, srv545318246=23, srv237831772=18, srv465487133=20, srv1350499850=3, srv1993005034=15, srv184755129=13, srv1582021464=10, srv782157877=27, srv1563172923=9, srv1427411420=6, srv1483419278=8, srv798693963=28, srv1205086099=2, srv1833622351=12, srv926741660=29, srv664507623=25, srv208000225=17, srv1861189694=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:53:19,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:53:19,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-09 19:53:19,600 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1219260205083117, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:53:19,600 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991257722817365, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1219260205083117, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-09 19:55:19,587 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5860593 different iterations. Found a solution that moves 1866 regions; Going from a computed imbalance of 0.991257722817365 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:55:19,594 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1061502750:90 , srv1078512979:90 , srv1205086099:90 , srv1350499850:90 , srv1415135134:90 , srv1417162432:90 , srv1427411420:90 , srv1430332521:90 , srv1483419278:90 , srv1563172923:90 , srv1582021464:90 , srv17392640:90 , srv1833622351:90 , srv184755129:90 , srv1861189694:90 , srv1993005034:90 , srv2052841128:90 , srv208000225:90 , srv237831772:90 , srv281256306:90 , srv465487133:90 , srv508181454:90 , srv537478003:90 , srv545318246:90 , srv661785619:90 , srv664507623:90 , srv74160577:90 , srv782157877:90 , srv798693963:90 , srv926741660:90 } 2024-01-09 19:55:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-09 19:55:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1078512979=1, srv1415135134=4, srv661785619=24, srv508181454=21, srv17392640=11, srv1061502750=0, srv1430332521=7, srv1417162432=5, srv2052841128=16, srv74160577=26, srv537478003=22, srv281256306=19, srv545318246=23, srv237831772=18, srv465487133=20, srv1350499850=3, srv1993005034=15, srv184755129=13, srv1582021464=10, srv782157877=27, srv1563172923=9, srv1427411420=6, srv1483419278=8, srv798693963=28, srv1205086099=2, srv1833622351=12, srv926741660=29, srv664507623=25, srv208000225=17, srv1861189694=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-09 19:55:19,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-09 19:55:19,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-09 19:55:19,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-09 19:55:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-09 19:55:19,619 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-09 19:55:19,619 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1061502750:90 , srv1078512979:90 , srv1205086099:90 , srv1350499850:90 , srv1415135134:90 , srv1417162432:90 , srv1427411420:90 , srv1430332521:90 , srv1483419278:90 , srv1563172923:90 , srv1582021464:90 , srv17392640:90 , srv1833622351:90 , srv184755129:90 , srv1861189694:90 , srv1993005034:90 , srv2052841128:90 , srv208000225:90 , srv237831772:90 , srv281256306:90 , srv465487133:90 , srv508181454:90 , srv537478003:90 , srv545318246:90 , srv661785619:90 , srv664507623:90 , srv74160577:90 , srv782157877:90 , srv798693963:90 , srv926741660:90 } 2024-01-09 19:55:19,634 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=261 (was 259) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=8697 (was 9548)