2023-05-27 06:53:22,833 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-27 06:53:23,121 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-27 06:53:23,227 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-27 06:53:23,288 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-27 06:53:23,295 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-27 06:53:23,308 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=297, ProcessCount=171, AvailableMemoryMB=5288 2023-05-27 06:53:23,310 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-27 06:53:23,311 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-27 06:53:23,438 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1028329888:84 , srv1051853349:84 , srv1063524177:84 , srv1102946254:84 , srv11090566:84 , srv1114791525:84 , srv1118483930:84 , srv1128135412:84 , srv114717027:84 , srv1162589043:84 , srv1184591664:84 , srv1188797182:84 , srv1204790034:84 , srv1225724445:84 , srv125440923:84 , srv1286128582:84 , srv128701076:84 , srv1348425356:84 , srv1356682424:84 , srv1378316078:84 , srv1389130311:84 , srv1401254548:84 , srv1405034585:84 , srv1406549255:84 , srv1424458270:84 , srv1449675481:84 , srv1450074370:84 , srv1507342114:84 , srv1533481254:84 , srv1538938531:84 , srv1608999018:84 , srv1618166925:84 , srv1618351779:84 , srv1680236412:84 , srv1709549658:84 , srv171695589:84 , srv1724141392:84 , srv1747512265:84 , srv1749323012:84 , srv1776050852:84 , srv1802435074:84 , srv1806350840:84 , srv1812110798:84 , srv1814628264:84 , srv1817599643:84 , srv1819258559:84 , srv182501247:84 , srv1825263489:84 , srv1831759893:84 , srv1832293988:84 , srv1839153418:84 , srv184769915:84 , srv1864617974:84 , srv1870037224:84 , srv1881492701:84 , srv188159477:84 , srv1907157028:84 , srv1913098785:84 , srv191461518:84 , srv1955385752:84 , srv1962338231:84 , srv1971301324:84 , srv197342751:84 , srv1984203667:84 , srv1994186130:84 , srv2011305735:84 , srv2014660942:84 , srv2024227653:84 , srv2058200704:84 , srv2062922713:84 , srv2080884132:84 , srv2083414160:84 , srv2092561822:84 , srv2145788012:84 , srv22913104:84 , srv237212768:84 , srv264116108:84 , srv299125776:84 , srv339858478:84 , srv351007015:84 , srv356784674:84 , srv386393879:84 , srv391982948:84 , srv393709218:84 , srv417972400:84 , srv426003785:84 , srv502050746:84 , srv508648309:84 , srv599026490:84 , srv716216401:84 , srv744468236:84 , srv748296983:84 , srv800323591:84 , srv824483808:84 , srv901295414:84 , srv9283733:84 , srv931517989:84 , srv934696017:84 , srv959355335:84 , srv535899795:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-27 06:53:23,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:23,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv959355335=99, srv351007015=79, srv1389130311=20, srv1724141392=36, srv1913098785=57, srv1881492701=54, srv2083414160=71, srv197342751=62, srv391982948=82, srv931517989=97, srv1028329888=0, srv1618166925=31, srv502050746=86, srv386393879=81, srv1538938531=29, srv171695589=35, srv184769915=51, srv1286128582=15, srv1102946254=3, srv1749323012=38, srv1984203667=63, srv1204790034=12, srv800323591=93, srv1962338231=60, srv599026490=89, srv125440923=14, srv1709549658=34, srv2024227653=67, srv237212768=75, srv1378316078=19, srv426003785=85, srv1128135412=7, srv1994186130=64, srv1063524177=2, srv299125776=77, srv356784674=80, srv191461518=58, srv1405034585=22, srv11090566=4, srv1618351779=32, srv1814628264=43, srv508648309=87, srv716216401=90, srv744468236=91, srv1533481254=28, srv1864617974=52, srv1955385752=59, srv2011305735=65, srv339858478=78, srv824483808=94, srv1225724445=13, srv1870037224=53, srv1356682424=18, srv1839153418=50, srv1971301324=61, srv1184591664=10, srv188159477=55, srv1449675481=25, srv1812110798=42, srv1831759893=48, srv1424458270=24, srv1747512265=37, srv9283733=96, srv114717027=8, srv1802435074=40, srv1817599643=44, srv1162589043=9, srv2080884132=70, srv748296983=92, srv393709218=83, srv1825263489=47, srv1819258559=45, srv2092561822=72, srv901295414=95, srv535899795=88, srv1188797182=11, srv22913104=74, srv1051853349=1, srv2062922713=69, srv1401254548=21, srv1907157028=56, srv1348425356=17, srv1776050852=39, srv934696017=98, srv1507342114=27, srv1832293988=49, srv1118483930=6, srv182501247=46, srv1406549255=23, srv1806350840=41, srv128701076=16, srv1680236412=33, srv1608999018=30, srv2145788012=73, srv264116108=76, srv2058200704=68, srv417972400=84, srv1450074370=26, srv2014660942=66, srv1114791525=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:23,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:23,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:23,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-27 06:53:23,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-27 06:53:23,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-27 06:53:23,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-27 06:53:23,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-27 06:53:23,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-27 06:53:23,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-27 06:53:23,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-27 06:53:23,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:23,708 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-27 06:53:23,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-27 06:53:24,045 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 492 ms to try 5000 different iterations. Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17835788605704953. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03913932509540508, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05374629682124858, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03913932509540508, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:24,064 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1348425356:82 , srv1984203667:82 , srv2080884132:82 , srv1028329888:83 , srv1450074370:83 , srv1962338231:83 , srv800323591:83 , srv1162589043:84 , srv1538938531:84 , srv1680236412:84 , srv1709549658:84 , srv1724141392:84 , srv1839153418:84 , srv1913098785:84 , srv1955385752:84 , srv2014660942:84 , srv824483808:84 , srv934696017:84 , srv1102946254:85 , srv1204790034:85 , srv1225724445:85 , srv125440923:85 , srv1286128582:85 , srv1507342114:85 , srv1533481254:85 , srv1618166925:85 , srv1618351779:85 , srv1812110798:85 , srv188159477:85 , srv191461518:85 , srv2011305735:85 , srv22913104:85 , srv237212768:85 , srv351007015:85 , srv386393879:85 , srv417972400:85 , srv744468236:85 , srv748296983:85 , srv1118483930:86 , srv1128135412:86 , srv1449675481:86 , srv1608999018:86 , srv1747512265:86 , srv1814628264:86 , srv182501247:86 , srv1832293988:86 , srv184769915:86 , srv1870037224:86 , srv1907157028:86 , srv508648309:86 , srv959355335:86 , srv1051853349:87 , srv11090566:87 , srv1405034585:87 , srv1749323012:87 , srv1817599643:87 , srv1819258559:87 , srv1825263489:87 , srv1881492701:87 , srv197342751:87 , srv2062922713:87 , srv2083414160:87 , srv2092561822:87 , srv2145788012:87 , srv339858478:87 , srv356784674:87 , srv393709218:87 , srv426003785:87 , srv502050746:87 , srv599026490:87 , srv716216401:87 , srv1114791525:88 , srv1356682424:88 , srv1378316078:88 , srv171695589:88 , srv1776050852:88 , srv1806350840:88 , srv1831759893:88 , srv2024227653:88 , srv901295414:88 , srv931517989:88 , srv1389130311:89 , srv1401254548:89 , srv1406549255:89 , srv1864617974:89 , srv1971301324:89 , srv1994186130:89 , srv264116108:89 , srv9283733:89 , srv1063524177:90 , srv128701076:90 , srv299125776:90 , srv114717027:91 , srv1184591664:91 , srv1188797182:91 , srv1802435074:91 , srv391982948:91 , srv2058200704:92 , srv1424458270:93 , srv535899795:438 } 2023-05-27 06:53:24,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:24,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv959355335=99, srv351007015=79, srv1389130311=20, srv1724141392=36, srv1913098785=57, srv1881492701=54, srv2083414160=71, srv197342751=62, srv391982948=82, srv931517989=97, srv1028329888=0, srv1618166925=31, srv502050746=86, srv386393879=81, srv1538938531=29, srv171695589=35, srv184769915=51, srv1286128582=15, srv1102946254=3, srv1749323012=38, srv1984203667=63, srv1204790034=12, srv800323591=93, srv1962338231=60, srv599026490=89, srv125440923=14, srv1709549658=34, srv2024227653=67, srv237212768=75, srv1378316078=19, srv426003785=85, srv1128135412=7, srv1994186130=64, srv1063524177=2, srv299125776=77, srv356784674=80, srv191461518=58, srv1405034585=22, srv11090566=4, srv1618351779=32, srv1814628264=43, srv508648309=87, srv716216401=90, srv744468236=91, srv1533481254=28, srv1864617974=52, srv1955385752=59, srv2011305735=65, srv339858478=78, srv824483808=94, srv1225724445=13, srv1870037224=53, srv1356682424=18, srv1839153418=50, srv1971301324=61, srv1184591664=10, srv188159477=55, srv1449675481=25, srv1812110798=42, srv1831759893=48, srv1424458270=24, srv1747512265=37, srv9283733=96, srv114717027=8, srv1802435074=40, srv1817599643=44, srv1162589043=9, srv2080884132=70, srv748296983=92, srv393709218=83, srv1825263489=47, srv1819258559=45, srv2092561822=72, srv901295414=95, srv535899795=88, srv1188797182=11, srv22913104=74, srv1051853349=1, srv2062922713=69, srv1401254548=21, srv1907157028=56, srv1348425356=17, srv1776050852=39, srv934696017=98, srv1507342114=27, srv1832293988=49, srv1118483930=6, srv182501247=46, srv1406549255=23, srv1806350840=41, srv128701076=16, srv1680236412=33, srv1608999018=30, srv2145788012=73, srv264116108=76, srv2058200704=68, srv417972400=84, srv1450074370=26, srv2014660942=66, srv1114791525=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:24,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-27 06:53:24,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-27 06:53:24,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-27 06:53:24,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-27 06:53:24,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-27 06:53:24,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-27 06:53:24,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-27 06:53:24,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-27 06:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-27 06:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-27 06:53:24,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-27 06:53:24,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03913932509540508, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05374629682124858, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03913932509540508, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:24,298 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-27 06:53:24,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1783250599794437, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03913932509540508, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05374629682124858, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03913932509540508, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-27 06:53:24,671 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 549 ms to try 5000 different iterations. Found a solution that moves 2651 regions; Going from a computed imbalance of 0.1783250599794437 to a new imbalance of 0.01512329318014292. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02216659704806667); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03352470318892807, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29455555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02216659704806667); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:24,683 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv114717027:78 , srv391982948:79 , srv1802435074:80 , srv1063524177:81 , srv1188797182:81 , srv1389130311:81 , srv1424458270:81 , srv128701076:82 , srv1776050852:82 , srv2058200704:82 , srv299125776:82 , srv1184591664:83 , srv1401254548:83 , srv171695589:83 , srv1971301324:83 , srv599026490:83 , srv901295414:83 , srv9283733:83 , srv1051853349:84 , srv11090566:84 , srv1114791525:84 , srv1128135412:84 , srv1356682424:84 , srv1378316078:84 , srv1825263489:84 , srv1831759893:84 , srv184769915:84 , srv2011305735:84 , srv2062922713:84 , srv2145788012:84 , srv502050746:84 , srv931517989:84 , srv1118483930:85 , srv1225724445:85 , srv1405034585:85 , srv1406549255:85 , srv1749323012:85 , srv1806350840:85 , srv1817599643:85 , srv182501247:85 , srv1994186130:85 , srv2083414160:85 , srv264116108:85 , srv339858478:85 , srv356784674:85 , srv393709218:85 , srv426003785:85 , srv748296983:85 , srv1102946254:86 , srv1204790034:86 , srv1449675481:86 , srv1507342114:86 , srv1608999018:86 , srv1812110798:86 , srv1819258559:86 , srv1832293988:86 , srv1870037224:86 , srv1881492701:86 , srv1907157028:86 , srv191461518:86 , srv2014660942:86 , srv22913104:86 , srv237212768:86 , srv508648309:86 , srv716216401:86 , srv959355335:86 , srv1162589043:87 , srv1286128582:87 , srv1533481254:87 , srv1618166925:87 , srv1618351779:87 , srv1709549658:87 , srv1724141392:87 , srv1839153418:87 , srv1913098785:87 , srv1955385752:87 , srv197342751:87 , srv2024227653:87 , srv386393879:87 , srv1747512265:88 , srv1814628264:88 , srv1864617974:88 , srv188159477:88 , srv351007015:88 , srv824483808:88 , srv125440923:89 , srv1450074370:89 , srv1538938531:89 , srv1680236412:89 , srv2080884132:89 , srv2092561822:89 , srv417972400:89 , srv744468236:89 , srv1028329888:90 , srv1348425356:90 , srv1962338231:90 , srv1984203667:90 , srv934696017:90 , srv800323591:92 , srv535899795:533 } 2023-05-27 06:53:24,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:24,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv959355335=99, srv351007015=79, srv1389130311=20, srv1724141392=36, srv1913098785=57, srv1881492701=54, srv2083414160=71, srv197342751=62, srv391982948=82, srv931517989=97, srv1028329888=0, srv1618166925=31, srv502050746=86, srv386393879=81, srv1538938531=29, srv171695589=35, srv184769915=51, srv1286128582=15, srv1102946254=3, srv1749323012=38, srv1984203667=63, srv1204790034=12, srv800323591=93, srv1962338231=60, srv599026490=89, srv125440923=14, srv1709549658=34, srv2024227653=67, srv237212768=75, srv1378316078=19, srv426003785=85, srv1128135412=7, srv1994186130=64, srv1063524177=2, srv299125776=77, srv356784674=80, srv191461518=58, srv1405034585=22, srv11090566=4, srv1618351779=32, srv1814628264=43, srv508648309=87, srv716216401=90, srv744468236=91, srv1533481254=28, srv1864617974=52, srv1955385752=59, srv2011305735=65, srv339858478=78, srv824483808=94, srv1225724445=13, srv1870037224=53, srv1356682424=18, srv1839153418=50, srv1971301324=61, srv1184591664=10, srv188159477=55, srv1449675481=25, srv1812110798=42, srv1831759893=48, srv1424458270=24, srv1747512265=37, srv9283733=96, srv114717027=8, srv1802435074=40, srv1817599643=44, srv1162589043=9, srv2080884132=70, srv748296983=92, srv393709218=83, srv1825263489=47, srv1819258559=45, srv2092561822=72, srv901295414=95, srv535899795=88, srv1188797182=11, srv22913104=74, srv1051853349=1, srv2062922713=69, srv1401254548=21, srv1907157028=56, srv1348425356=17, srv1776050852=39, srv934696017=98, srv1507342114=27, srv1832293988=49, srv1118483930=6, srv182501247=46, srv1406549255=23, srv1806350840=41, srv128701076=16, srv1680236412=33, srv1608999018=30, srv2145788012=73, srv264116108=76, srv2058200704=68, srv417972400=84, srv1450074370=26, srv2014660942=66, srv1114791525=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:24,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:24,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:24,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:24,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-27 06:53:24,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-27 06:53:24,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-27 06:53:24,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-27 06:53:24,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-27 06:53:24,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-27 06:53:24,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-27 06:53:24,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-27 06:53:24,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-27 06:53:24,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-27 06:53:24,875 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02216659704806667); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03352470318892807, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02216659704806667); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:24,875 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-27 06:53:24,875 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015104730481394404, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02216659704806667); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03352470318892807, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02216659704806667); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-27 06:53:25,139 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 375 ms to try 5000 different iterations. Found a solution that moves 1032 regions; Going from a computed imbalance of 0.015104730481394404 to a new imbalance of 8.049014052372184E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052091796918016745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00663291117550788); MoveCostFunction : (multiplier=7.0, imbalance=0.11466666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052091796918016745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:25,141 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1864617974:81 , srv2092561822:81 , srv1406549255:83 , srv1814628264:83 , srv2024227653:83 , srv2058200704:83 , srv264116108:83 , srv417972400:83 , srv800323591:83 , srv1028329888:84 , srv1184591664:84 , srv125440923:84 , srv1424458270:84 , srv1680236412:84 , srv1747512265:84 , srv188159477:84 , srv1907157028:84 , srv1971301324:84 , srv197342751:84 , srv1994186130:84 , srv744468236:84 , srv1114791525:85 , srv1286128582:85 , srv128701076:85 , srv1348425356:85 , srv1356682424:85 , srv1401254548:85 , srv1405034585:85 , srv1449675481:85 , srv1538938531:85 , srv1608999018:85 , srv1618166925:85 , srv1618351779:85 , srv1806350840:85 , srv1819258559:85 , srv1870037224:85 , srv1881492701:85 , srv1962338231:85 , srv351007015:85 , srv386393879:85 , srv426003785:85 , srv716216401:85 , srv9283733:85 , srv931517989:85 , srv934696017:85 , srv1051853349:86 , srv1063524177:86 , srv11090566:86 , srv1118483930:86 , srv1162589043:86 , srv1188797182:86 , srv1378316078:86 , srv1450074370:86 , srv1507342114:86 , srv1533481254:86 , srv1709549658:86 , srv1724141392:86 , srv1749323012:86 , srv1802435074:86 , srv1812110798:86 , srv1817599643:86 , srv1831759893:86 , srv1832293988:86 , srv1913098785:86 , srv1984203667:86 , srv2080884132:86 , srv22913104:86 , srv299125776:86 , srv339858478:86 , srv356784674:86 , srv393709218:86 , srv502050746:86 , srv508648309:86 , srv959355335:86 , srv1102946254:87 , srv1204790034:87 , srv171695589:87 , srv1776050852:87 , srv182501247:87 , srv1825263489:87 , srv1839153418:87 , srv184769915:87 , srv191461518:87 , srv2014660942:87 , srv2083414160:87 , srv2145788012:87 , srv237212768:87 , srv391982948:87 , srv599026490:87 , srv748296983:87 , srv824483808:87 , srv901295414:87 , srv1128135412:88 , srv114717027:88 , srv1225724445:88 , srv1389130311:88 , srv1955385752:88 , srv2011305735:88 , srv2062922713:88 , srv535899795:533 } 2023-05-27 06:53:25,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:25,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv959355335=99, srv351007015=79, srv1389130311=20, srv1724141392=36, srv1913098785=57, srv1881492701=54, srv2083414160=71, srv197342751=62, srv391982948=82, srv931517989=97, srv1028329888=0, srv1618166925=31, srv502050746=86, srv386393879=81, srv1538938531=29, srv171695589=35, srv184769915=51, srv1286128582=15, srv1102946254=3, srv1749323012=38, srv1984203667=63, srv1204790034=12, srv800323591=93, srv1962338231=60, srv599026490=89, srv125440923=14, srv1709549658=34, srv2024227653=67, srv237212768=75, srv1378316078=19, srv426003785=85, srv1128135412=7, srv1994186130=64, srv1063524177=2, srv299125776=77, srv356784674=80, srv191461518=58, srv1405034585=22, srv11090566=4, srv1618351779=32, srv1814628264=43, srv508648309=87, srv716216401=90, srv744468236=91, srv1533481254=28, srv1864617974=52, srv1955385752=59, srv2011305735=65, srv339858478=78, srv824483808=94, srv1225724445=13, srv1870037224=53, srv1356682424=18, srv1839153418=50, srv1971301324=61, srv1184591664=10, srv188159477=55, srv1449675481=25, srv1812110798=42, srv1831759893=48, srv1424458270=24, srv1747512265=37, srv9283733=96, srv114717027=8, srv1802435074=40, srv1817599643=44, srv1162589043=9, srv2080884132=70, srv748296983=92, srv393709218=83, srv1825263489=47, srv1819258559=45, srv2092561822=72, srv901295414=95, srv535899795=88, srv1188797182=11, srv22913104=74, srv1051853349=1, srv2062922713=69, srv1401254548=21, srv1907157028=56, srv1348425356=17, srv1776050852=39, srv934696017=98, srv1507342114=27, srv1832293988=49, srv1118483930=6, srv182501247=46, srv1406549255=23, srv1806350840=41, srv128701076=16, srv1680236412=33, srv1608999018=30, srv2145788012=73, srv264116108=76, srv2058200704=68, srv417972400=84, srv1450074370=26, srv2014660942=66, srv1114791525=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:25,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:25,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:25,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:25,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-27 06:53:25,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-27 06:53:25,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-27 06:53:25,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-27 06:53:25,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-27 06:53:25,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-27 06:53:25,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-27 06:53:25,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-27 06:53:25,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-27 06:53:25,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-27 06:53:25,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-27 06:53:25,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-27 06:53:25,295 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052091796918016745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00663291117550788); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052091796918016745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:25,296 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-27 06:53:25,296 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.976751867881545E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052091796918016745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00663291117550788); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052091796918016745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-27 06:53:25,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 482 ms to try 5000 different iterations. Found a solution that moves 236 regions; Going from a computed imbalance of 7.976751867881545E-4 to a new imbalance of 2.7855688602435167E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.026222222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:25,669 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1204790034:83 , srv1533481254:83 , srv1955385752:83 , srv2062922713:83 , srv2083414160:83 , srv356784674:83 , srv824483808:83 , srv934696017:83 , srv1102946254:84 , srv11090566:84 , srv1128135412:84 , srv1162589043:84 , srv1188797182:84 , srv1225724445:84 , srv1378316078:84 , srv1389130311:84 , srv1424458270:84 , srv1450074370:84 , srv1538938531:84 , srv1618351779:84 , srv1709549658:84 , srv171695589:84 , srv1747512265:84 , srv1749323012:84 , srv1802435074:84 , srv1806350840:84 , srv1817599643:84 , srv1819258559:84 , srv1825263489:84 , srv1831759893:84 , srv1832293988:84 , srv1839153418:84 , srv1864617974:84 , srv1881492701:84 , srv1962338231:84 , srv197342751:84 , srv1984203667:84 , srv1994186130:84 , srv2024227653:84 , srv2145788012:84 , srv237212768:84 , srv299125776:84 , srv339858478:84 , srv393709218:84 , srv417972400:84 , srv508648309:84 , srv716216401:84 , srv744468236:84 , srv748296983:84 , srv800323591:84 , srv901295414:84 , srv959355335:84 , srv1028329888:85 , srv1051853349:85 , srv1063524177:85 , srv1114791525:85 , srv1118483930:85 , srv114717027:85 , srv1184591664:85 , srv125440923:85 , srv1286128582:85 , srv128701076:85 , srv1348425356:85 , srv1401254548:85 , srv1405034585:85 , srv1406549255:85 , srv1449675481:85 , srv1507342114:85 , srv1608999018:85 , srv1618166925:85 , srv1680236412:85 , srv1724141392:85 , srv1812110798:85 , srv1814628264:85 , srv182501247:85 , srv184769915:85 , srv1870037224:85 , srv188159477:85 , srv1913098785:85 , srv191461518:85 , srv2011305735:85 , srv2014660942:85 , srv2058200704:85 , srv2080884132:85 , srv2092561822:85 , srv264116108:85 , srv351007015:85 , srv386393879:85 , srv426003785:85 , srv502050746:85 , srv599026490:85 , srv9283733:85 , srv931517989:85 , srv1356682424:86 , srv1776050852:86 , srv1907157028:86 , srv22913104:86 , srv391982948:86 , srv1971301324:87 , srv535899795:638 } 2023-05-27 06:53:25,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:25,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv959355335=99, srv351007015=79, srv1389130311=20, srv1724141392=36, srv1913098785=57, srv1881492701=54, srv2083414160=71, srv197342751=62, srv391982948=82, srv931517989=97, srv1028329888=0, srv1618166925=31, srv502050746=86, srv386393879=81, srv1538938531=29, srv171695589=35, srv184769915=51, srv1286128582=15, srv1102946254=3, srv1749323012=38, srv1984203667=63, srv1204790034=12, srv800323591=93, srv1962338231=60, srv599026490=89, srv125440923=14, srv1709549658=34, srv2024227653=67, srv237212768=75, srv1378316078=19, srv426003785=85, srv1128135412=7, srv1994186130=64, srv1063524177=2, srv299125776=77, srv356784674=80, srv191461518=58, srv1405034585=22, srv11090566=4, srv1618351779=32, srv1814628264=43, srv508648309=87, srv716216401=90, srv744468236=91, srv1533481254=28, srv1864617974=52, srv1955385752=59, srv2011305735=65, srv339858478=78, srv824483808=94, srv1225724445=13, srv1870037224=53, srv1356682424=18, srv1839153418=50, srv1971301324=61, srv1184591664=10, srv188159477=55, srv1449675481=25, srv1812110798=42, srv1831759893=48, srv1424458270=24, srv1747512265=37, srv9283733=96, srv114717027=8, srv1802435074=40, srv1817599643=44, srv1162589043=9, srv2080884132=70, srv748296983=92, srv393709218=83, srv1825263489=47, srv1819258559=45, srv2092561822=72, srv901295414=95, srv535899795=88, srv1188797182=11, srv22913104=74, srv1051853349=1, srv2062922713=69, srv1401254548=21, srv1907157028=56, srv1348425356=17, srv1776050852=39, srv934696017=98, srv1507342114=27, srv1832293988=49, srv1118483930=6, srv182501247=46, srv1406549255=23, srv1806350840=41, srv128701076=16, srv1680236412=33, srv1608999018=30, srv2145788012=73, srv264116108=76, srv2058200704=68, srv417972400=84, srv1450074370=26, srv2014660942=66, srv1114791525=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:25,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:25,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-27 06:53:25,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-27 06:53:25,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-27 06:53:25,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-27 06:53:25,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-27 06:53:25,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-27 06:53:25,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-27 06:53:25,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-27 06:53:25,796 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:25,797 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-27 06:53:25,797 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6203181282688005E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-27 06:53:26,040 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 339 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 2.6203181282688005E-5 to a new imbalance of 6.940129244310425E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:26,040 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1184591664:83 , srv125440923:83 , srv1356682424:83 , srv1776050852:83 , srv182501247:83 , srv191461518:83 , srv1971301324:83 , srv22913104:83 , srv351007015:83 , srv391982948:83 , srv1028329888:84 , srv1051853349:84 , srv1063524177:84 , srv1102946254:84 , srv1114791525:84 , srv1118483930:84 , srv1128135412:84 , srv114717027:84 , srv1188797182:84 , srv1225724445:84 , srv1286128582:84 , srv128701076:84 , srv1348425356:84 , srv1378316078:84 , srv1389130311:84 , srv1401254548:84 , srv1405034585:84 , srv1406549255:84 , srv1424458270:84 , srv1449675481:84 , srv1450074370:84 , srv1507342114:84 , srv1538938531:84 , srv1608999018:84 , srv1618166925:84 , srv1680236412:84 , srv171695589:84 , srv1724141392:84 , srv1747512265:84 , srv1749323012:84 , srv1806350840:84 , srv1812110798:84 , srv1814628264:84 , srv1817599643:84 , srv1819258559:84 , srv1825263489:84 , srv1831759893:84 , srv1832293988:84 , srv1839153418:84 , srv184769915:84 , srv1864617974:84 , srv1870037224:84 , srv1881492701:84 , srv188159477:84 , srv1907157028:84 , srv1913098785:84 , srv1955385752:84 , srv1962338231:84 , srv197342751:84 , srv1984203667:84 , srv1994186130:84 , srv2011305735:84 , srv2014660942:84 , srv2024227653:84 , srv2058200704:84 , srv2062922713:84 , srv2080884132:84 , srv2083414160:84 , srv2092561822:84 , srv2145788012:84 , srv237212768:84 , srv264116108:84 , srv299125776:84 , srv339858478:84 , srv386393879:84 , srv393709218:84 , srv426003785:84 , srv502050746:84 , srv508648309:84 , srv599026490:84 , srv716216401:84 , srv744468236:84 , srv800323591:84 , srv824483808:84 , srv901295414:84 , srv9283733:84 , srv931517989:84 , srv934696017:84 , srv959355335:84 , srv11090566:85 , srv1162589043:85 , srv1204790034:85 , srv1533481254:85 , srv1618351779:85 , srv1709549658:85 , srv1802435074:85 , srv356784674:85 , srv417972400:85 , srv748296983:85 , srv535899795:684 } 2023-05-27 06:53:26,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:26,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv959355335=99, srv351007015=79, srv1389130311=20, srv1724141392=36, srv1913098785=57, srv1881492701=54, srv2083414160=71, srv197342751=62, srv391982948=82, srv931517989=97, srv1028329888=0, srv1618166925=31, srv502050746=86, srv386393879=81, srv1538938531=29, srv171695589=35, srv184769915=51, srv1286128582=15, srv1102946254=3, srv1749323012=38, srv1984203667=63, srv1204790034=12, srv800323591=93, srv1962338231=60, srv599026490=89, srv125440923=14, srv1709549658=34, srv2024227653=67, srv237212768=75, srv1378316078=19, srv426003785=85, srv1128135412=7, srv1994186130=64, srv1063524177=2, srv299125776=77, srv356784674=80, srv191461518=58, srv1405034585=22, srv11090566=4, srv1618351779=32, srv1814628264=43, srv508648309=87, srv716216401=90, srv744468236=91, srv1533481254=28, srv1864617974=52, srv1955385752=59, srv2011305735=65, srv339858478=78, srv824483808=94, srv1225724445=13, srv1870037224=53, srv1356682424=18, srv1839153418=50, srv1971301324=61, srv1184591664=10, srv188159477=55, srv1449675481=25, srv1812110798=42, srv1831759893=48, srv1424458270=24, srv1747512265=37, srv9283733=96, srv114717027=8, srv1802435074=40, srv1817599643=44, srv1162589043=9, srv2080884132=70, srv748296983=92, srv393709218=83, srv1825263489=47, srv1819258559=45, srv2092561822=72, srv901295414=95, srv535899795=88, srv1188797182=11, srv22913104=74, srv1051853349=1, srv2062922713=69, srv1401254548=21, srv1907157028=56, srv1348425356=17, srv1776050852=39, srv934696017=98, srv1507342114=27, srv1832293988=49, srv1118483930=6, srv182501247=46, srv1406549255=23, srv1806350840=41, srv128701076=16, srv1680236412=33, srv1608999018=30, srv2145788012=73, srv264116108=76, srv2058200704=68, srv417972400=84, srv1450074370=26, srv2014660942=66, srv1114791525=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:26,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-27 06:53:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-27 06:53:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-27 06:53:26,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-27 06:53:26,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-27 06:53:26,167 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:26,168 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1184591664:83 , srv125440923:83 , srv1356682424:83 , srv1776050852:83 , srv182501247:83 , srv191461518:83 , srv1971301324:83 , srv22913104:83 , srv351007015:83 , srv391982948:83 , srv1028329888:84 , srv1051853349:84 , srv1063524177:84 , srv1102946254:84 , srv1114791525:84 , srv1118483930:84 , srv1128135412:84 , srv114717027:84 , srv1188797182:84 , srv1225724445:84 , srv1286128582:84 , srv128701076:84 , srv1348425356:84 , srv1378316078:84 , srv1389130311:84 , srv1401254548:84 , srv1405034585:84 , srv1406549255:84 , srv1424458270:84 , srv1449675481:84 , srv1450074370:84 , srv1507342114:84 , srv1538938531:84 , srv1608999018:84 , srv1618166925:84 , srv1680236412:84 , srv171695589:84 , srv1724141392:84 , srv1747512265:84 , srv1749323012:84 , srv1806350840:84 , srv1812110798:84 , srv1814628264:84 , srv1817599643:84 , srv1819258559:84 , srv1825263489:84 , srv1831759893:84 , srv1832293988:84 , srv1839153418:84 , srv184769915:84 , srv1864617974:84 , srv1870037224:84 , srv1881492701:84 , srv188159477:84 , srv1907157028:84 , srv1913098785:84 , srv1955385752:84 , srv1962338231:84 , srv197342751:84 , srv1984203667:84 , srv1994186130:84 , srv2011305735:84 , srv2014660942:84 , srv2024227653:84 , srv2058200704:84 , srv2062922713:84 , srv2080884132:84 , srv2083414160:84 , srv2092561822:84 , srv2145788012:84 , srv237212768:84 , srv264116108:84 , srv299125776:84 , srv339858478:84 , srv386393879:84 , srv393709218:84 , srv426003785:84 , srv502050746:84 , srv508648309:84 , srv599026490:84 , srv716216401:84 , srv744468236:84 , srv800323591:84 , srv824483808:84 , srv901295414:84 , srv9283733:84 , srv931517989:84 , srv934696017:84 , srv959355335:84 , srv11090566:85 , srv1162589043:85 , srv1204790034:85 , srv1533481254:85 , srv1618351779:85 , srv1709549658:85 , srv1802435074:85 , srv356784674:85 , srv417972400:85 , srv748296983:85 , srv535899795:684 } 2023-05-27 06:53:26,191 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=313 (was 297) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=4817 (was 5288) 2023-05-27 06:53:26,198 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=313, ProcessCount=171, AvailableMemoryMB=4817 2023-05-27 06:53:26,198 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-27 06:53:26,199 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-27 06:53:26,205 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1099622285:84 , srv1104920579:84 , srv125015596:84 , srv1464738115:84 , srv1563257933:84 , srv1664266169:84 , srv1703144985:84 , srv1794896112:84 , srv1988071114:84 , srv2014375797:84 , srv2021091603:84 , srv2021323532:84 , srv2099918310:84 , srv226593970:84 , srv271467943:84 , srv280273070:84 , srv454630111:84 , srv466787276:84 , srv485761061:84 , srv524217357:84 , srv554808789:84 , srv558370403:84 , srv582384502:84 , srv657627482:84 , srv751776904:84 , srv796886990:84 , srv822819784:84 , srv91383964:84 , srv989560458:84 , srv260222866:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-27 06:53:26,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:53:26,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2099918310=12, srv1794896112=7, srv271467943=15, srv1099622285=0, srv466787276=18, srv1703144985=6, srv454630111=17, srv1464738115=3, srv1563257933=4, srv1664266169=5, srv226593970=13, srv657627482=24, srv2021091603=10, srv582384502=23, srv524217357=20, srv558370403=22, srv125015596=2, srv260222866=14, srv989560458=29, srv2021323532=11, srv2014375797=9, srv485761061=19, srv554808789=21, srv822819784=27, srv796886990=26, srv280273070=16, srv1104920579=1, srv1988071114=8, srv91383964=28, srv751776904=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:53:26,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:53:26,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-27 06:53:26,231 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1358831280689181, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:53:26,232 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912621206563832, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1358831280689181, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-27 06:55:26,220 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5820272 different iterations. Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912621206563832 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:55:26,229 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1099622285:90 , srv1104920579:90 , srv125015596:90 , srv1464738115:90 , srv1563257933:90 , srv1664266169:90 , srv1703144985:90 , srv1794896112:90 , srv1988071114:90 , srv2014375797:90 , srv2021091603:90 , srv2021323532:90 , srv2099918310:90 , srv226593970:90 , srv260222866:90 , srv271467943:90 , srv280273070:90 , srv454630111:90 , srv466787276:90 , srv485761061:90 , srv524217357:90 , srv554808789:90 , srv558370403:90 , srv582384502:90 , srv657627482:90 , srv751776904:90 , srv796886990:90 , srv822819784:90 , srv91383964:90 , srv989560458:90 } 2023-05-27 06:55:26,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-27 06:55:26,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2099918310=12, srv1794896112=7, srv271467943=15, srv1099622285=0, srv466787276=18, srv1703144985=6, srv454630111=17, srv1464738115=3, srv1563257933=4, srv1664266169=5, srv226593970=13, srv657627482=24, srv2021091603=10, srv582384502=23, srv524217357=20, srv558370403=22, srv125015596=2, srv260222866=14, srv989560458=29, srv2021323532=11, srv2014375797=9, srv485761061=19, srv554808789=21, srv822819784=27, srv796886990=26, srv280273070=16, srv1104920579=1, srv1988071114=8, srv91383964=28, srv751776904=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-27 06:55:26,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-27 06:55:26,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-27 06:55:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-27 06:55:26,250 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-27 06:55:26,250 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1099622285:90 , srv1104920579:90 , srv125015596:90 , srv1464738115:90 , srv1563257933:90 , srv1664266169:90 , srv1703144985:90 , srv1794896112:90 , srv1988071114:90 , srv2014375797:90 , srv2021091603:90 , srv2021323532:90 , srv2099918310:90 , srv226593970:90 , srv260222866:90 , srv271467943:90 , srv280273070:90 , srv454630111:90 , srv466787276:90 , srv485761061:90 , srv524217357:90 , srv554808789:90 , srv558370403:90 , srv582384502:90 , srv657627482:90 , srv751776904:90 , srv796886990:90 , srv822819784:90 , srv91383964:90 , srv989560458:90 } 2023-05-27 06:55:26,262 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=262 (was 313), ProcessCount=169 (was 171), AvailableMemoryMB=4667 (was 4817)