2023-10-23 00:53:28,036 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-23 00:53:28,353 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-23 00:53:28,467 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-23 00:53:28,535 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-23 00:53:28,542 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-23 00:53:28,555 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=259, ProcessCount=169, AvailableMemoryMB=8460 2023-10-23 00:53:28,556 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-23 00:53:28,557 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-23 00:53:28,699 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1053698894:84 , srv1072808336:84 , srv1093562705:84 , srv1123874500:84 , srv1125539081:84 , srv1132414915:84 , srv1137662012:84 , srv1158513802:84 , srv1160485707:84 , srv1202758533:84 , srv1239603957:84 , srv1262041924:84 , srv1265288579:84 , srv1270797665:84 , srv1278050370:84 , srv1279070255:84 , srv129088550:84 , srv1291181766:84 , srv131493773:84 , srv1322984924:84 , srv1331036195:84 , srv1351605662:84 , srv1360475306:84 , srv1381922662:84 , srv1420126038:84 , srv1426823392:84 , srv1431473727:84 , srv1440321861:84 , srv1462813719:84 , srv1483245708:84 , srv1483459352:84 , srv1485319321:84 , srv1536071575:84 , srv1560802215:84 , srv1642673239:84 , srv1644593246:84 , srv1664330596:84 , srv1664424753:84 , srv1669332201:84 , srv1769709745:84 , srv180468545:84 , srv1813463654:84 , srv1824610598:84 , srv1828294428:84 , srv1828683718:84 , srv1864027229:84 , srv1877392288:84 , srv1904797602:84 , srv1912298509:84 , srv191763371:84 , srv1929378697:84 , srv1954445756:84 , srv1985714635:84 , srv2013511813:84 , srv2031859888:84 , srv2082053100:84 , srv208870616:84 , srv2107112667:84 , srv211071872:84 , srv2127045901:84 , srv218260931:84 , srv224562586:84 , srv229222320:84 , srv243964712:84 , srv257238832:84 , srv309523534:84 , srv317646691:84 , srv342570001:84 , srv356929730:84 , srv39874055:84 , srv413504974:84 , srv468391693:84 , srv498384406:84 , srv505247905:84 , srv533011404:84 , srv57091027:84 , srv583724843:84 , srv599582719:84 , srv604871905:84 , srv608666823:84 , srv615502556:84 , srv635091940:84 , srv636022600:84 , srv662329175:84 , srv717696931:84 , srv738981276:84 , srv75185091:84 , srv75528587:84 , srv788542963:84 , srv802825236:84 , srv811390186:84 , srv815302711:84 , srv825092848:84 , srv825926746:84 , srv827782312:84 , srv874284181:84 , srv903193182:84 , srv912957042:84 , srv953280291:84 , srv1194632085:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-23 00:53:28,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:28,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1331036195=21, srv1912298509=49, srv180468545=41, srv1202758533=10, srv1536071575=33, srv211071872=59, srv1669332201=39, srv243964712=64, srv1279070255=16, srv1160485707=8, srv1360475306=23, srv309523534=66, srv599582719=78, srv1824610598=43, srv1462813719=29, srv1132414915=5, srv75185091=87, srv2031859888=55, srv1483245708=30, srv1270797665=14, srv131493773=19, srv468391693=72, srv1137662012=6, srv583724843=77, srv802825236=90, srv1239603957=11, srv717696931=85, srv903193182=97, srv208870616=57, srv1194632085=9, srv604871905=79, srv1426823392=26, srv615502556=81, srv1560802215=34, srv1828683718=45, srv224562586=62, srv1877392288=47, srv1351605662=22, srv1072808336=1, srv1278050370=15, srv129088550=17, srv1864027229=46, srv257238832=65, srv191763371=50, srv533011404=75, srv1483459352=31, srv1929378697=51, srv57091027=76, srv413504974=71, srv1642673239=35, srv75528587=88, srv498384406=73, srv811390186=91, srv2127045901=60, srv1123874500=3, srv2107112667=58, srv636022600=83, srv317646691=67, srv953280291=99, srv874284181=96, srv1262041924=12, srv912957042=98, srv1904797602=48, srv1381922662=24, srv1420126038=25, srv1644593246=36, srv1291181766=18, srv39874055=70, srv788542963=89, srv635091940=82, srv815302711=92, srv1158513802=7, srv1954445756=52, srv1125539081=4, srv1664424753=38, srv1265288579=13, srv827782312=95, srv825092848=93, srv1322984924=20, srv1440321861=28, srv608666823=80, srv1664330596=37, srv1769709745=40, srv218260931=61, srv1485319321=32, srv342570001=68, srv505247905=74, srv1093562705=2, srv2082053100=56, srv1985714635=53, srv229222320=63, srv1813463654=42, srv1053698894=0, srv662329175=84, srv738981276=86, srv825926746=94, srv1431473727=27, srv2013511813=54, srv356929730=69, srv1828294428=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:28,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:28,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:28,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:28,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:28,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-23 00:53:28,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-23 00:53:28,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-23 00:53:28,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-23 00:53:28,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-23 00:53:28,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-23 00:53:28,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-23 00:53:28,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-23 00:53:28,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-23 00:53:28,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-23 00:53:28,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-23 00:53:28,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-23 00:53:28,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-23 00:53:28,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-23 00:53:28,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-23 00:53:28,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-23 00:53:28,990 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:28,990 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-23 00:53:28,991 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-23 00:53:29,311 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 471 ms to try 5000 different iterations. Found a solution that moves 4684 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17939029088410638. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04053114127610748, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05328280434678001, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5204444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04053114127610748, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17458333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24158333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:29,321 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv635091940:79 , srv1642673239:80 , srv802825236:80 , srv825926746:80 , srv1160485707:81 , srv1560802215:81 , srv229222320:81 , srv1158513802:82 , srv1440321861:83 , srv1912298509:83 , srv1954445756:83 , srv342570001:83 , srv505247905:83 , srv1125539081:84 , srv1322984924:84 , srv1483459352:84 , srv1485319321:84 , srv1644593246:84 , srv1669332201:84 , srv1824610598:84 , srv2107112667:84 , srv257238832:84 , srv39874055:84 , srv825092848:84 , srv1270797665:85 , srv1278050370:85 , srv180468545:85 , srv1828294428:85 , srv2013511813:85 , srv2127045901:85 , srv243964712:85 , srv498384406:85 , srv608666823:85 , srv738981276:85 , srv912957042:85 , srv1053698894:86 , srv1132414915:86 , srv1262041924:86 , srv131493773:86 , srv1331036195:86 , srv1462813719:86 , srv1769709745:86 , srv1813463654:86 , srv191763371:86 , srv1985714635:86 , srv224562586:86 , srv636022600:86 , srv75528587:86 , srv1093562705:87 , srv1381922662:87 , srv1420126038:87 , srv1431473727:87 , srv1483245708:87 , srv1664330596:87 , srv1664424753:87 , srv1904797602:87 , srv2082053100:87 , srv208870616:87 , srv317646691:87 , srv599582719:87 , srv615502556:87 , srv811390186:87 , srv1123874500:88 , srv1137662012:88 , srv1239603957:88 , srv1291181766:88 , srv1351605662:88 , srv1536071575:88 , srv1877392288:88 , srv1929378697:88 , srv218260931:88 , srv309523534:88 , srv356929730:88 , srv413504974:88 , srv468391693:88 , srv57091027:88 , srv662329175:88 , srv788542963:88 , srv815302711:88 , srv827782312:88 , srv903193182:88 , srv953280291:88 , srv1202758533:89 , srv1265288579:89 , srv1279070255:89 , srv129088550:89 , srv1360475306:89 , srv1828683718:89 , srv1864027229:89 , srv211071872:89 , srv533011404:89 , srv75185091:89 , srv1072808336:90 , srv2031859888:90 , srv604871905:91 , srv1426823392:92 , srv717696931:93 , srv874284181:94 , srv583724843:96 , srv1194632085:450 } 2023-10-23 00:53:29,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:29,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1331036195=21, srv1912298509=49, srv180468545=41, srv1202758533=10, srv1536071575=33, srv211071872=59, srv1669332201=39, srv243964712=64, srv1279070255=16, srv1160485707=8, srv1360475306=23, srv309523534=66, srv599582719=78, srv1824610598=43, srv1462813719=29, srv1132414915=5, srv75185091=87, srv2031859888=55, srv1483245708=30, srv1270797665=14, srv131493773=19, srv468391693=72, srv1137662012=6, srv583724843=77, srv802825236=90, srv1239603957=11, srv717696931=85, srv903193182=97, srv208870616=57, srv1194632085=9, srv604871905=79, srv1426823392=26, srv615502556=81, srv1560802215=34, srv1828683718=45, srv224562586=62, srv1877392288=47, srv1351605662=22, srv1072808336=1, srv1278050370=15, srv129088550=17, srv1864027229=46, srv257238832=65, srv191763371=50, srv533011404=75, srv1483459352=31, srv1929378697=51, srv57091027=76, srv413504974=71, srv1642673239=35, srv75528587=88, srv498384406=73, srv811390186=91, srv2127045901=60, srv1123874500=3, srv2107112667=58, srv636022600=83, srv317646691=67, srv953280291=99, srv874284181=96, srv1262041924=12, srv912957042=98, srv1904797602=48, srv1381922662=24, srv1420126038=25, srv1644593246=36, srv1291181766=18, srv39874055=70, srv788542963=89, srv635091940=82, srv815302711=92, srv1158513802=7, srv1954445756=52, srv1125539081=4, srv1664424753=38, srv1265288579=13, srv827782312=95, srv825092848=93, srv1322984924=20, srv1440321861=28, srv608666823=80, srv1664330596=37, srv1769709745=40, srv218260931=61, srv1485319321=32, srv342570001=68, srv505247905=74, srv1093562705=2, srv2082053100=56, srv1985714635=53, srv229222320=63, srv1813463654=42, srv1053698894=0, srv662329175=84, srv738981276=86, srv825926746=94, srv1431473727=27, srv2013511813=54, srv356929730=69, srv1828294428=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:29,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:29,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:29,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:29,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-23 00:53:29,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-23 00:53:29,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-23 00:53:29,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-23 00:53:29,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-23 00:53:29,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-23 00:53:29,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-23 00:53:29,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-23 00:53:29,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-23 00:53:29,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04053114127610748, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05328280434678001, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04053114127610748, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17458333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24158333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:29,538 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-23 00:53:29,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17935749281509922, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04053114127610748, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05328280434678001, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04053114127610748, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17458333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24158333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-23 00:53:29,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 418 ms to try 5000 different iterations. Found a solution that moves 2644 regions; Going from a computed imbalance of 0.17935749281509922 to a new imbalance of 0.013278620936201717. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022409424735970615); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03347444877124522, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2937777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022409424735970615); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:29,821 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv583724843:77 , srv717696931:78 , srv874284181:79 , srv1291181766:80 , srv1426823392:80 , srv1864027229:80 , srv211071872:80 , srv1279070255:81 , srv1360475306:81 , srv2031859888:81 , srv413504974:81 , srv1072808336:82 , srv1929378697:82 , srv604871905:82 , srv1265288579:83 , srv129088550:83 , srv1351605662:83 , srv1877392288:83 , srv218260931:83 , srv533011404:83 , srv599582719:83 , srv615502556:83 , srv788542963:83 , srv827782312:83 , srv1093562705:84 , srv1123874500:84 , srv1202758533:84 , srv1239603957:84 , srv1664330596:84 , srv1664424753:84 , srv1769709745:84 , srv1813463654:84 , srv1904797602:84 , srv309523534:84 , srv498384406:84 , srv811390186:84 , srv1053698894:85 , srv1278050370:85 , srv1536071575:85 , srv1828683718:85 , srv191763371:85 , srv317646691:85 , srv356929730:85 , srv636022600:85 , srv75185091:85 , srv815302711:85 , srv1270797665:86 , srv1331036195:86 , srv1420126038:86 , srv1462813719:86 , srv1485319321:86 , srv1828294428:86 , srv2082053100:86 , srv208870616:86 , srv257238832:86 , srv468391693:86 , srv662329175:86 , srv738981276:86 , srv903193182:86 , srv1125539081:87 , srv1137662012:87 , srv131493773:87 , srv1322984924:87 , srv1381922662:87 , srv1431473727:87 , srv180468545:87 , srv1954445756:87 , srv2107112667:87 , srv224562586:87 , srv75528587:87 , srv953280291:87 , srv1132414915:88 , srv1262041924:88 , srv1669332201:88 , srv1824610598:88 , srv1985714635:88 , srv2127045901:88 , srv243964712:88 , srv39874055:88 , srv505247905:88 , srv912957042:88 , srv1483245708:89 , srv1483459352:89 , srv2013511813:89 , srv57091027:89 , srv608666823:89 , srv1440321861:90 , srv1644593246:90 , srv1912298509:90 , srv229222320:90 , srv342570001:90 , srv635091940:90 , srv825092848:90 , srv825926746:90 , srv1158513802:91 , srv1160485707:91 , srv1560802215:91 , srv1642673239:92 , srv802825236:93 , srv1194632085:523 } 2023-10-23 00:53:29,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:29,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1331036195=21, srv1912298509=49, srv180468545=41, srv1202758533=10, srv1536071575=33, srv211071872=59, srv1669332201=39, srv243964712=64, srv1279070255=16, srv1160485707=8, srv1360475306=23, srv309523534=66, srv599582719=78, srv1824610598=43, srv1462813719=29, srv1132414915=5, srv75185091=87, srv2031859888=55, srv1483245708=30, srv1270797665=14, srv131493773=19, srv468391693=72, srv1137662012=6, srv583724843=77, srv802825236=90, srv1239603957=11, srv717696931=85, srv903193182=97, srv208870616=57, srv1194632085=9, srv604871905=79, srv1426823392=26, srv615502556=81, srv1560802215=34, srv1828683718=45, srv224562586=62, srv1877392288=47, srv1351605662=22, srv1072808336=1, srv1278050370=15, srv129088550=17, srv1864027229=46, srv257238832=65, srv191763371=50, srv533011404=75, srv1483459352=31, srv1929378697=51, srv57091027=76, srv413504974=71, srv1642673239=35, srv75528587=88, srv498384406=73, srv811390186=91, srv2127045901=60, srv1123874500=3, srv2107112667=58, srv636022600=83, srv317646691=67, srv953280291=99, srv874284181=96, srv1262041924=12, srv912957042=98, srv1904797602=48, srv1381922662=24, srv1420126038=25, srv1644593246=36, srv1291181766=18, srv39874055=70, srv788542963=89, srv635091940=82, srv815302711=92, srv1158513802=7, srv1954445756=52, srv1125539081=4, srv1664424753=38, srv1265288579=13, srv827782312=95, srv825092848=93, srv1322984924=20, srv1440321861=28, srv608666823=80, srv1664330596=37, srv1769709745=40, srv218260931=61, srv1485319321=32, srv342570001=68, srv505247905=74, srv1093562705=2, srv2082053100=56, srv1985714635=53, srv229222320=63, srv1813463654=42, srv1053698894=0, srv662329175=84, srv738981276=86, srv825926746=94, srv1431473727=27, srv2013511813=54, srv356929730=69, srv1828294428=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:29,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:29,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:29,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-23 00:53:29,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-23 00:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-23 00:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-23 00:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-23 00:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-23 00:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-23 00:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-23 00:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-23 00:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-23 00:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-23 00:53:30,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022409424735970615); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03347444877124522, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022409424735970615); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:30,002 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-23 00:53:30,003 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01326010725250082, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022409424735970615); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03347444877124522, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022409424735970615); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-23 00:53:30,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 450 ms to try 5000 different iterations. Found a solution that moves 928 regions; Going from a computed imbalance of 0.01326010725250082 to a new imbalance of 9.027674645388105E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010657403385139376); MoveCostFunction : (multiplier=7.0, imbalance=0.10311111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007899480725663495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:30,344 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv57091027:80 , srv1483245708:81 , srv1985714635:82 , srv1132414915:83 , srv1137662012:83 , srv1381922662:83 , srv1431473727:83 , srv1828683718:83 , srv468391693:83 , srv608666823:83 , srv662329175:83 , srv802825236:83 , srv903193182:83 , srv953280291:83 , srv1158513802:84 , srv1262041924:84 , srv1265288579:84 , srv1440321861:84 , srv1536071575:84 , srv1912298509:84 , srv2082053100:84 , srv208870616:84 , srv2127045901:84 , srv243964712:84 , srv342570001:84 , srv75185091:84 , srv815302711:84 , srv825092848:84 , srv1053698894:85 , srv1072808336:85 , srv1093562705:85 , srv1160485707:85 , srv1202758533:85 , srv129088550:85 , srv131493773:85 , srv1420126038:85 , srv1426823392:85 , srv1462813719:85 , srv1644593246:85 , srv1669332201:85 , srv180468545:85 , srv1877392288:85 , srv2013511813:85 , srv2107112667:85 , srv224562586:85 , srv309523534:85 , srv356929730:85 , srv39874055:85 , srv533011404:85 , srv583724843:85 , srv75528587:85 , srv874284181:85 , srv1123874500:86 , srv1239603957:86 , srv1270797665:86 , srv1279070255:86 , srv1322984924:86 , srv1331036195:86 , srv1483459352:86 , srv1485319321:86 , srv1560802215:86 , srv1642673239:86 , srv1664424753:86 , srv1824610598:86 , srv1828294428:86 , srv218260931:86 , srv317646691:86 , srv505247905:86 , srv599582719:86 , srv604871905:86 , srv636022600:86 , srv738981276:86 , srv788542963:86 , srv825926746:86 , srv827782312:86 , srv912957042:86 , srv1125539081:87 , srv1278050370:87 , srv1351605662:87 , srv1360475306:87 , srv1664330596:87 , srv1769709745:87 , srv1813463654:87 , srv1864027229:87 , srv1904797602:87 , srv191763371:87 , srv1929378697:87 , srv1954445756:87 , srv2031859888:87 , srv229222320:87 , srv257238832:87 , srv615502556:87 , srv717696931:87 , srv811390186:87 , srv211071872:88 , srv413504974:88 , srv1291181766:89 , srv498384406:89 , srv635091940:89 , srv1194632085:555 } 2023-10-23 00:53:30,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:30,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1331036195=21, srv1912298509=49, srv180468545=41, srv1202758533=10, srv1536071575=33, srv211071872=59, srv1669332201=39, srv243964712=64, srv1279070255=16, srv1160485707=8, srv1360475306=23, srv309523534=66, srv599582719=78, srv1824610598=43, srv1462813719=29, srv1132414915=5, srv75185091=87, srv2031859888=55, srv1483245708=30, srv1270797665=14, srv131493773=19, srv468391693=72, srv1137662012=6, srv583724843=77, srv802825236=90, srv1239603957=11, srv717696931=85, srv903193182=97, srv208870616=57, srv1194632085=9, srv604871905=79, srv1426823392=26, srv615502556=81, srv1560802215=34, srv1828683718=45, srv224562586=62, srv1877392288=47, srv1351605662=22, srv1072808336=1, srv1278050370=15, srv129088550=17, srv1864027229=46, srv257238832=65, srv191763371=50, srv533011404=75, srv1483459352=31, srv1929378697=51, srv57091027=76, srv413504974=71, srv1642673239=35, srv75528587=88, srv498384406=73, srv811390186=91, srv2127045901=60, srv1123874500=3, srv2107112667=58, srv636022600=83, srv317646691=67, srv953280291=99, srv874284181=96, srv1262041924=12, srv912957042=98, srv1904797602=48, srv1381922662=24, srv1420126038=25, srv1644593246=36, srv1291181766=18, srv39874055=70, srv788542963=89, srv635091940=82, srv815302711=92, srv1158513802=7, srv1954445756=52, srv1125539081=4, srv1664424753=38, srv1265288579=13, srv827782312=95, srv825092848=93, srv1322984924=20, srv1440321861=28, srv608666823=80, srv1664330596=37, srv1769709745=40, srv218260931=61, srv1485319321=32, srv342570001=68, srv505247905=74, srv1093562705=2, srv2082053100=56, srv1985714635=53, srv229222320=63, srv1813463654=42, srv1053698894=0, srv662329175=84, srv738981276=86, srv825926746=94, srv1431473727=27, srv2013511813=54, srv356929730=69, srv1828294428=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:30,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:30,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:30,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-23 00:53:30,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-23 00:53:30,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-23 00:53:30,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-23 00:53:30,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-23 00:53:30,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-23 00:53:30,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-23 00:53:30,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-23 00:53:30,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-23 00:53:30,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-23 00:53:30,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-23 00:53:30,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-23 00:53:30,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-23 00:53:30,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010657403385139376); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007899480725663495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:30,501 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-23 00:53:30,502 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.962694696543808E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010657403385139376); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007899480725663495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-23 00:53:30,842 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 266 regions; Going from a computed imbalance of 8.962694696543808E-4 to a new imbalance of 4.1122745060219416E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.029555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:30,843 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1483459352:83 , srv1644593246:83 , srv2013511813:83 , srv498384406:83 , srv604871905:83 , srv635091940:83 , srv912957042:83 , srv1123874500:84 , srv1125539081:84 , srv1137662012:84 , srv1202758533:84 , srv1239603957:84 , srv129088550:84 , srv131493773:84 , srv1331036195:84 , srv1351605662:84 , srv1420126038:84 , srv1560802215:84 , srv1642673239:84 , srv1664330596:84 , srv1824610598:84 , srv1904797602:84 , srv191763371:84 , srv208870616:84 , srv224562586:84 , srv229222320:84 , srv317646691:84 , srv356929730:84 , srv583724843:84 , srv717696931:84 , srv75185091:84 , srv75528587:84 , srv788542963:84 , srv811390186:84 , srv825092848:84 , srv874284181:84 , srv903193182:84 , srv953280291:84 , srv1072808336:85 , srv1132414915:85 , srv1158513802:85 , srv1160485707:85 , srv1262041924:85 , srv1270797665:85 , srv1278050370:85 , srv1291181766:85 , srv1322984924:85 , srv1360475306:85 , srv1381922662:85 , srv1426823392:85 , srv1431473727:85 , srv1440321861:85 , srv1462813719:85 , srv1483245708:85 , srv1536071575:85 , srv1664424753:85 , srv1669332201:85 , srv1769709745:85 , srv180468545:85 , srv1813463654:85 , srv1828294428:85 , srv1828683718:85 , srv1912298509:85 , srv1929378697:85 , srv1954445756:85 , srv2031859888:85 , srv2082053100:85 , srv211071872:85 , srv2127045901:85 , srv218260931:85 , srv243964712:85 , srv257238832:85 , srv309523534:85 , srv342570001:85 , srv39874055:85 , srv468391693:85 , srv505247905:85 , srv533011404:85 , srv57091027:85 , srv615502556:85 , srv636022600:85 , srv662329175:85 , srv738981276:85 , srv827782312:85 , srv1053698894:86 , srv1093562705:86 , srv1265288579:86 , srv1279070255:86 , srv1485319321:86 , srv1864027229:86 , srv1985714635:86 , srv2107112667:86 , srv413504974:86 , srv599582719:86 , srv608666823:86 , srv802825236:86 , srv815302711:86 , srv825926746:86 , srv1877392288:87 , srv1194632085:614 } 2023-10-23 00:53:30,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:30,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1331036195=21, srv1912298509=49, srv180468545=41, srv1202758533=10, srv1536071575=33, srv211071872=59, srv1669332201=39, srv243964712=64, srv1279070255=16, srv1160485707=8, srv1360475306=23, srv309523534=66, srv599582719=78, srv1824610598=43, srv1462813719=29, srv1132414915=5, srv75185091=87, srv2031859888=55, srv1483245708=30, srv1270797665=14, srv131493773=19, srv468391693=72, srv1137662012=6, srv583724843=77, srv802825236=90, srv1239603957=11, srv717696931=85, srv903193182=97, srv208870616=57, srv1194632085=9, srv604871905=79, srv1426823392=26, srv615502556=81, srv1560802215=34, srv1828683718=45, srv224562586=62, srv1877392288=47, srv1351605662=22, srv1072808336=1, srv1278050370=15, srv129088550=17, srv1864027229=46, srv257238832=65, srv191763371=50, srv533011404=75, srv1483459352=31, srv1929378697=51, srv57091027=76, srv413504974=71, srv1642673239=35, srv75528587=88, srv498384406=73, srv811390186=91, srv2127045901=60, srv1123874500=3, srv2107112667=58, srv636022600=83, srv317646691=67, srv953280291=99, srv874284181=96, srv1262041924=12, srv912957042=98, srv1904797602=48, srv1381922662=24, srv1420126038=25, srv1644593246=36, srv1291181766=18, srv39874055=70, srv788542963=89, srv635091940=82, srv815302711=92, srv1158513802=7, srv1954445756=52, srv1125539081=4, srv1664424753=38, srv1265288579=13, srv827782312=95, srv825092848=93, srv1322984924=20, srv1440321861=28, srv608666823=80, srv1664330596=37, srv1769709745=40, srv218260931=61, srv1485319321=32, srv342570001=68, srv505247905=74, srv1093562705=2, srv2082053100=56, srv1985714635=53, srv229222320=63, srv1813463654=42, srv1053698894=0, srv662329175=84, srv738981276=86, srv825926746=94, srv1431473727=27, srv2013511813=54, srv356929730=69, srv1828294428=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:30,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-23 00:53:30,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-23 00:53:30,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-23 00:53:30,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-23 00:53:30,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-23 00:53:30,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-23 00:53:30,974 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:30,975 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-23 00:53:30,975 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.9260173250673885E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-23 00:53:31,236 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 364 ms to try 5000 different iterations. Found a solution that moves 21 regions; Going from a computed imbalance of 3.9260173250673885E-5 to a new imbalance of 6.8911141966908056E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0023333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:31,236 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1262041924:83 , srv1877392288:83 , srv2031859888:83 , srv413504974:83 , srv608666823:83 , srv815302711:83 , srv1053698894:84 , srv1072808336:84 , srv1093562705:84 , srv1123874500:84 , srv1125539081:84 , srv1132414915:84 , srv1137662012:84 , srv1158513802:84 , srv1160485707:84 , srv1202758533:84 , srv1239603957:84 , srv1265288579:84 , srv1270797665:84 , srv1278050370:84 , srv1279070255:84 , srv1291181766:84 , srv131493773:84 , srv1322984924:84 , srv1331036195:84 , srv1351605662:84 , srv1360475306:84 , srv1381922662:84 , srv1420126038:84 , srv1426823392:84 , srv1431473727:84 , srv1440321861:84 , srv1462813719:84 , srv1483245708:84 , srv1483459352:84 , srv1485319321:84 , srv1536071575:84 , srv1560802215:84 , srv1642673239:84 , srv1644593246:84 , srv1664330596:84 , srv1664424753:84 , srv1669332201:84 , srv1769709745:84 , srv180468545:84 , srv1813463654:84 , srv1824610598:84 , srv1828294428:84 , srv1828683718:84 , srv1864027229:84 , srv1904797602:84 , srv1912298509:84 , srv191763371:84 , srv1929378697:84 , srv1954445756:84 , srv1985714635:84 , srv2013511813:84 , srv2082053100:84 , srv2107112667:84 , srv211071872:84 , srv2127045901:84 , srv218260931:84 , srv224562586:84 , srv229222320:84 , srv243964712:84 , srv257238832:84 , srv309523534:84 , srv317646691:84 , srv342570001:84 , srv356929730:84 , srv39874055:84 , srv468391693:84 , srv505247905:84 , srv533011404:84 , srv57091027:84 , srv583724843:84 , srv599582719:84 , srv604871905:84 , srv615502556:84 , srv636022600:84 , srv662329175:84 , srv717696931:84 , srv738981276:84 , srv75185091:84 , srv75528587:84 , srv802825236:84 , srv811390186:84 , srv825092848:84 , srv825926746:84 , srv827782312:84 , srv874284181:84 , srv912957042:84 , srv953280291:84 , srv129088550:85 , srv208870616:85 , srv498384406:85 , srv635091940:85 , srv788542963:85 , srv903193182:85 , srv1194632085:684 } 2023-10-23 00:53:31,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:31,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1331036195=21, srv1912298509=49, srv180468545=41, srv1202758533=10, srv1536071575=33, srv211071872=59, srv1669332201=39, srv243964712=64, srv1279070255=16, srv1160485707=8, srv1360475306=23, srv309523534=66, srv599582719=78, srv1824610598=43, srv1462813719=29, srv1132414915=5, srv75185091=87, srv2031859888=55, srv1483245708=30, srv1270797665=14, srv131493773=19, srv468391693=72, srv1137662012=6, srv583724843=77, srv802825236=90, srv1239603957=11, srv717696931=85, srv903193182=97, srv208870616=57, srv1194632085=9, srv604871905=79, srv1426823392=26, srv615502556=81, srv1560802215=34, srv1828683718=45, srv224562586=62, srv1877392288=47, srv1351605662=22, srv1072808336=1, srv1278050370=15, srv129088550=17, srv1864027229=46, srv257238832=65, srv191763371=50, srv533011404=75, srv1483459352=31, srv1929378697=51, srv57091027=76, srv413504974=71, srv1642673239=35, srv75528587=88, srv498384406=73, srv811390186=91, srv2127045901=60, srv1123874500=3, srv2107112667=58, srv636022600=83, srv317646691=67, srv953280291=99, srv874284181=96, srv1262041924=12, srv912957042=98, srv1904797602=48, srv1381922662=24, srv1420126038=25, srv1644593246=36, srv1291181766=18, srv39874055=70, srv788542963=89, srv635091940=82, srv815302711=92, srv1158513802=7, srv1954445756=52, srv1125539081=4, srv1664424753=38, srv1265288579=13, srv827782312=95, srv825092848=93, srv1322984924=20, srv1440321861=28, srv608666823=80, srv1664330596=37, srv1769709745=40, srv218260931=61, srv1485319321=32, srv342570001=68, srv505247905=74, srv1093562705=2, srv2082053100=56, srv1985714635=53, srv229222320=63, srv1813463654=42, srv1053698894=0, srv662329175=84, srv738981276=86, srv825926746=94, srv1431473727=27, srv2013511813=54, srv356929730=69, srv1828294428=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:31,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:31,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-23 00:53:31,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-23 00:53:31,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-23 00:53:31,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-23 00:53:31,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-23 00:53:31,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-23 00:53:31,388 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:31,389 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1262041924:83 , srv1877392288:83 , srv2031859888:83 , srv413504974:83 , srv608666823:83 , srv815302711:83 , srv1053698894:84 , srv1072808336:84 , srv1093562705:84 , srv1123874500:84 , srv1125539081:84 , srv1132414915:84 , srv1137662012:84 , srv1158513802:84 , srv1160485707:84 , srv1202758533:84 , srv1239603957:84 , srv1265288579:84 , srv1270797665:84 , srv1278050370:84 , srv1279070255:84 , srv1291181766:84 , srv131493773:84 , srv1322984924:84 , srv1331036195:84 , srv1351605662:84 , srv1360475306:84 , srv1381922662:84 , srv1420126038:84 , srv1426823392:84 , srv1431473727:84 , srv1440321861:84 , srv1462813719:84 , srv1483245708:84 , srv1483459352:84 , srv1485319321:84 , srv1536071575:84 , srv1560802215:84 , srv1642673239:84 , srv1644593246:84 , srv1664330596:84 , srv1664424753:84 , srv1669332201:84 , srv1769709745:84 , srv180468545:84 , srv1813463654:84 , srv1824610598:84 , srv1828294428:84 , srv1828683718:84 , srv1864027229:84 , srv1904797602:84 , srv1912298509:84 , srv191763371:84 , srv1929378697:84 , srv1954445756:84 , srv1985714635:84 , srv2013511813:84 , srv2082053100:84 , srv2107112667:84 , srv211071872:84 , srv2127045901:84 , srv218260931:84 , srv224562586:84 , srv229222320:84 , srv243964712:84 , srv257238832:84 , srv309523534:84 , srv317646691:84 , srv342570001:84 , srv356929730:84 , srv39874055:84 , srv468391693:84 , srv505247905:84 , srv533011404:84 , srv57091027:84 , srv583724843:84 , srv599582719:84 , srv604871905:84 , srv615502556:84 , srv636022600:84 , srv662329175:84 , srv717696931:84 , srv738981276:84 , srv75185091:84 , srv75528587:84 , srv802825236:84 , srv811390186:84 , srv825092848:84 , srv825926746:84 , srv827782312:84 , srv874284181:84 , srv912957042:84 , srv953280291:84 , srv129088550:85 , srv208870616:85 , srv498384406:85 , srv635091940:85 , srv788542963:85 , srv903193182:85 , srv1194632085:684 } 2023-10-23 00:53:31,420 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=286 (was 259) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7951 (was 8460) 2023-10-23 00:53:31,429 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=286, ProcessCount=169, AvailableMemoryMB=7951 2023-10-23 00:53:31,430 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-23 00:53:31,430 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-23 00:53:31,438 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1130464767:84 , srv1209421834:84 , srv1233716280:84 , srv1278792540:84 , srv1305321859:84 , srv1306858516:84 , srv1355075064:84 , srv1412453503:84 , srv1413949777:84 , srv1419838150:84 , srv1615854200:84 , srv1676240023:84 , srv1794653061:84 , srv1912256049:84 , srv1963174916:84 , srv2030408341:84 , srv2036623822:84 , srv2132577217:84 , srv2143587772:84 , srv2146891915:84 , srv4495920:84 , srv472517720:84 , srv497883467:84 , srv549720708:84 , srv73482296:84 , srv791258683:84 , srv871484722:84 , srv877839443:84 , srv951093814:84 , srv840116331:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-23 00:53:31,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:53:31,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2036623822=16, srv2132577217=17, srv951093814=29, srv1412453503=7, srv472517720=21, srv791258683=25, srv1615854200=10, srv1306858516=5, srv840116331=26, srv871484722=27, srv549720708=23, srv877839443=28, srv1912256049=13, srv2030408341=15, srv73482296=24, srv1305321859=4, srv2143587772=18, srv1413949777=8, srv1130464767=0, srv497883467=22, srv1233716280=2, srv1676240023=11, srv1963174916=14, srv2146891915=19, srv1209421834=1, srv1355075064=6, srv4495920=20, srv1278792540=3, srv1794653061=12, srv1419838150=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:53:31,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:53:31,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-23 00:53:31,465 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.094917042208561, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:53:31,465 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912492123764952, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.094917042208561, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-23 00:55:31,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6223495 different iterations. Found a solution that moves 1873 regions; Going from a computed imbalance of 0.9912492123764952 to a new imbalance of 4.3716754376927055E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6937037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:55:31,460 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1130464767:90 , srv1209421834:90 , srv1233716280:90 , srv1278792540:90 , srv1305321859:90 , srv1306858516:90 , srv1355075064:90 , srv1412453503:90 , srv1413949777:90 , srv1419838150:90 , srv1615854200:90 , srv1676240023:90 , srv1794653061:90 , srv1912256049:90 , srv1963174916:90 , srv2030408341:90 , srv2036623822:90 , srv2132577217:90 , srv2143587772:90 , srv2146891915:90 , srv4495920:90 , srv472517720:90 , srv497883467:90 , srv549720708:90 , srv73482296:90 , srv791258683:90 , srv840116331:90 , srv871484722:90 , srv877839443:90 , srv951093814:90 } 2023-10-23 00:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-23 00:55:31,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2036623822=16, srv2132577217=17, srv951093814=29, srv1412453503=7, srv472517720=21, srv791258683=25, srv1615854200=10, srv1306858516=5, srv840116331=26, srv871484722=27, srv549720708=23, srv877839443=28, srv1912256049=13, srv2030408341=15, srv73482296=24, srv1305321859=4, srv2143587772=18, srv1413949777=8, srv1130464767=0, srv497883467=22, srv1233716280=2, srv1676240023=11, srv1963174916=14, srv2146891915=19, srv1209421834=1, srv1355075064=6, srv4495920=20, srv1278792540=3, srv1794653061=12, srv1419838150=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-23 00:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-23 00:55:31,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-23 00:55:31,483 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-23 00:55:31,484 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1130464767:90 , srv1209421834:90 , srv1233716280:90 , srv1278792540:90 , srv1305321859:90 , srv1306858516:90 , srv1355075064:90 , srv1412453503:90 , srv1413949777:90 , srv1419838150:90 , srv1615854200:90 , srv1676240023:90 , srv1794653061:90 , srv1912256049:90 , srv1963174916:90 , srv2030408341:90 , srv2036623822:90 , srv2132577217:90 , srv2143587772:90 , srv2146891915:90 , srv4495920:90 , srv472517720:90 , srv497883467:90 , srv549720708:90 , srv73482296:90 , srv791258683:90 , srv840116331:90 , srv871484722:90 , srv877839443:90 , srv951093814:90 } 2023-10-23 00:55:31,501 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=174 (was 286), ProcessCount=167 (was 169), AvailableMemoryMB=7811 (was 7951)