2024-02-28 01:53:16,197 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-28 01:53:16,530 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-28 01:53:16,653 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-28 01:53:16,727 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-28 01:53:16,738 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-28 01:53:16,754 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=172, AvailableMemoryMB=11331 2024-02-28 01:53:16,756 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-28 01:53:16,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-28 01:53:16,884 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001017253:84 , srv1024015885:84 , srv1074622705:84 , srv1077996948:84 , srv109490929:84 , srv1130993062:84 , srv1136059698:84 , srv1147026089:84 , srv1177935051:84 , srv1194824963:84 , srv1195562931:84 , srv1265525689:84 , srv1270354324:84 , srv1283223431:84 , srv1310356261:84 , srv131782702:84 , srv1320972383:84 , srv1368066385:84 , srv1377898458:84 , srv13968576:84 , srv140461060:84 , srv1412774548:84 , srv1420979409:84 , srv1435924811:84 , srv1440779696:84 , srv145964263:84 , srv1466920157:84 , srv1478390128:84 , srv1531265960:84 , srv1531853618:84 , srv1538325089:84 , srv1567741296:84 , srv1575347272:84 , srv157744273:84 , srv1580094725:84 , srv1588658112:84 , srv1624353758:84 , srv1647051074:84 , srv168744652:84 , srv1721279471:84 , srv1745033871:84 , srv1749632160:84 , srv176505309:84 , srv1767334513:84 , srv178346533:84 , srv1795110190:84 , srv1802588064:84 , srv182252191:84 , srv1847450513:84 , srv1866416769:84 , srv1866808472:84 , srv1888826320:84 , srv1891145529:84 , srv193985876:84 , srv1954313090:84 , srv1958339209:84 , srv2002498539:84 , srv201701921:84 , srv2017743610:84 , srv2028781987:84 , srv2078024885:84 , srv2090235659:84 , srv2090796402:84 , srv2105149974:84 , srv2110360545:84 , srv229815984:84 , srv254356802:84 , srv268733977:84 , srv289480482:84 , srv321699843:84 , srv332131412:84 , srv368109864:84 , srv394939:84 , srv405227456:84 , srv474768671:84 , srv476619947:84 , srv489158087:84 , srv50652414:84 , srv53057353:84 , srv544370942:84 , srv556232993:84 , srv558737949:84 , srv562273542:84 , srv580001053:84 , srv612231144:84 , srv618247639:84 , srv650192161:84 , srv669617599:84 , srv681950842:84 , srv724117192:84 , srv735858498:84 , srv737623164:84 , srv738672848:84 , srv741221188:84 , srv764034905:84 , srv781843073:84 , srv865920488:84 , srv968523712:84 , srv973873837:84 , srv918556553:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-28 01:53:16,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:16,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1412774548=21, srv2090235659=61, srv1538325089=30, srv157744273=33, srv1588658112=35, srv2105149974=63, srv1265525689=11, srv741221188=93, srv2078024885=60, srv544370942=79, srv168744652=38, srv193985876=53, srv1136059698=6, srv178346533=44, srv1767334513=43, srv1795110190=45, srv1866808472=50, srv558737949=81, srv2028781987=59, srv724117192=89, srv1624353758=36, srv1891145529=52, srv1575347272=32, srv1531265960=28, srv1077996948=3, srv368109864=71, srv1177935051=8, srv781843073=95, srv176505309=42, srv650192161=86, srv229815984=65, srv2110360545=64, srv1147026089=7, srv1074622705=2, srv1866416769=49, srv737623164=91, srv618247639=85, srv1888826320=51, srv145964263=25, srv254356802=66, srv562273542=82, srv764034905=94, srv556232993=80, srv109490929=4, srv53057353=78, srv1310356261=14, srv1420979409=22, srv140461060=20, srv681950842=88, srv1802588064=46, srv865920488=96, srv1435924811=23, srv268733977=67, srv474768671=74, srv1440779696=24, srv1466920157=26, srv405227456=73, srv1001017253=0, srv332131412=70, srv289480482=68, srv13968576=19, srv2090796402=62, srv1531853618=29, srv50652414=77, srv1958339209=55, srv1647051074=37, srv201701921=57, srv1195562931=10, srv2002498539=56, srv1749632160=41, srv580001053=83, srv1320972383=16, srv321699843=69, srv394939=72, srv973873837=99, srv1270354324=12, srv1954313090=54, srv1478390128=27, srv1024015885=1, srv182252191=47, srv1283223431=13, srv1580094725=34, srv1847450513=48, srv476619947=75, srv2017743610=58, srv612231144=84, srv1567741296=31, srv735858498=90, srv738672848=92, srv918556553=97, srv968523712=98, srv131782702=15, srv1377898458=18, srv1368066385=17, srv1721279471=39, srv489158087=76, srv1194824963=9, srv669617599=87, srv1745033871=40, srv1130993062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:16,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:16,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:16,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:16,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-28 01:53:16,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-28 01:53:16,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-28 01:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-28 01:53:16,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-28 01:53:16,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-28 01:53:16,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-28 01:53:16,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-28 01:53:16,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-28 01:53:16,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-28 01:53:16,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-28 01:53:16,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-28 01:53:16,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-28 01:53:16,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-28 01:53:16,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-28 01:53:16,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-28 01:53:16,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-28 01:53:16,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-28 01:53:16,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-28 01:53:16,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-28 01:53:16,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-28 01:53:16,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-28 01:53:16,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-28 01:53:16,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-28 01:53:16,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-28 01:53:16,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-28 01:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-28 01:53:16,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-28 01:53:16,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-28 01:53:16,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-28 01:53:16,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-28 01:53:16,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-28 01:53:16,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-28 01:53:16,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-28 01:53:16,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-28 01:53:16,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-28 01:53:16,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-28 01:53:16,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-28 01:53:16,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-28 01:53:16,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-28 01:53:17,227 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:17,228 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-28 01:53:17,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-28 01:53:18,217 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1190 ms to try 5000 different iterations. Found a solution that moves 4677 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18344091837367857. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04108577494529849, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05923000119471867, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5196666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04108577494529849, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17883333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:18,247 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2090796402:79 , srv131782702:80 , srv1420979409:80 , srv1749632160:80 , srv1466920157:82 , srv168744652:82 , srv2017743610:82 , srv2028781987:82 , srv1024015885:83 , srv1074622705:83 , srv1195562931:83 , srv368109864:83 , srv556232993:83 , srv650192161:83 , srv1478390128:84 , srv1567741296:84 , srv1580094725:84 , srv193985876:84 , srv1954313090:84 , srv394939:84 , srv405227456:84 , srv474768671:84 , srv53057353:84 , srv735858498:84 , srv764034905:84 , srv973873837:84 , srv1177935051:85 , srv1270354324:85 , srv1310356261:85 , srv13968576:85 , srv1588658112:85 , srv182252191:85 , srv1866808472:85 , srv1891145529:85 , srv2002498539:85 , srv558737949:85 , srv669617599:85 , srv737623164:85 , srv865920488:85 , srv1265525689:86 , srv1412774548:86 , srv1440779696:86 , srv157744273:86 , srv178346533:86 , srv1888826320:86 , srv2105149974:86 , srv2110360545:86 , srv254356802:86 , srv332131412:86 , srv580001053:86 , srv738672848:86 , srv781843073:86 , srv968523712:86 , srv1136059698:87 , srv1283223431:87 , srv176505309:87 , srv1802588064:87 , srv1958339209:87 , srv229815984:87 , srv289480482:87 , srv618247639:87 , srv681950842:87 , srv741221188:87 , srv1001017253:88 , srv1147026089:88 , srv1320972383:88 , srv1368066385:88 , srv1377898458:88 , srv140461060:88 , srv1531853618:88 , srv1538325089:88 , srv1721279471:88 , srv1767334513:88 , srv1847450513:88 , srv268733977:88 , srv476619947:88 , srv489158087:88 , srv612231144:88 , srv724117192:88 , srv1077996948:89 , srv109490929:89 , srv1624353758:89 , srv201701921:89 , srv50652414:89 , srv544370942:89 , srv562273542:89 , srv1130993062:90 , srv1194824963:90 , srv145964263:90 , srv1647051074:90 , srv1745033871:90 , srv1795110190:90 , srv321699843:90 , srv1575347272:91 , srv1435924811:92 , srv1531265960:92 , srv1866416769:92 , srv2090235659:92 , srv2078024885:93 , srv918556553:455 } 2024-02-28 01:53:18,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:18,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1412774548=21, srv2090235659=61, srv1538325089=30, srv157744273=33, srv1588658112=35, srv2105149974=63, srv1265525689=11, srv741221188=93, srv2078024885=60, srv544370942=79, srv168744652=38, srv193985876=53, srv1136059698=6, srv178346533=44, srv1767334513=43, srv1795110190=45, srv1866808472=50, srv558737949=81, srv2028781987=59, srv724117192=89, srv1624353758=36, srv1891145529=52, srv1575347272=32, srv1531265960=28, srv1077996948=3, srv368109864=71, srv1177935051=8, srv781843073=95, srv176505309=42, srv650192161=86, srv229815984=65, srv2110360545=64, srv1147026089=7, srv1074622705=2, srv1866416769=49, srv737623164=91, srv618247639=85, srv1888826320=51, srv145964263=25, srv254356802=66, srv562273542=82, srv764034905=94, srv556232993=80, srv109490929=4, srv53057353=78, srv1310356261=14, srv1420979409=22, srv140461060=20, srv681950842=88, srv1802588064=46, srv865920488=96, srv1435924811=23, srv268733977=67, srv474768671=74, srv1440779696=24, srv1466920157=26, srv405227456=73, srv1001017253=0, srv332131412=70, srv289480482=68, srv13968576=19, srv2090796402=62, srv1531853618=29, srv50652414=77, srv1958339209=55, srv1647051074=37, srv201701921=57, srv1195562931=10, srv2002498539=56, srv1749632160=41, srv580001053=83, srv1320972383=16, srv321699843=69, srv394939=72, srv973873837=99, srv1270354324=12, srv1954313090=54, srv1478390128=27, srv1024015885=1, srv182252191=47, srv1283223431=13, srv1580094725=34, srv1847450513=48, srv476619947=75, srv2017743610=58, srv612231144=84, srv1567741296=31, srv735858498=90, srv738672848=92, srv918556553=97, srv968523712=98, srv131782702=15, srv1377898458=18, srv1368066385=17, srv1721279471=39, srv489158087=76, srv1194824963=9, srv669617599=87, srv1745033871=40, srv1130993062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:18,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:18,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:18,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:18,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-28 01:53:18,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-28 01:53:18,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-28 01:53:18,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-28 01:53:18,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-28 01:53:18,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-28 01:53:18,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-28 01:53:18,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-28 01:53:18,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-28 01:53:18,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-28 01:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-28 01:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-28 01:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-28 01:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-28 01:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-28 01:53:18,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04108577494529849, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05923000119471867, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04108577494529849, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17883333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:18,475 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-28 01:53:18,476 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.183408169319719, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04108577494529849, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05923000119471867, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04108577494529849, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17883333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-28 01:53:18,823 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations. Found a solution that moves 2618 regions; Going from a computed imbalance of 0.183408169319719 to a new imbalance of 0.016564667702439632. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024442403751161725); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03801470564291989, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29088888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024442403751161725); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:18,836 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2090235659:77 , srv1575347272:78 , srv1866416769:79 , srv321699843:79 , srv1130993062:80 , srv201701921:80 , srv2078024885:80 , srv1001017253:81 , srv1435924811:81 , srv1745033871:81 , srv1795110190:81 , srv50652414:81 , srv1368066385:82 , srv145964263:82 , srv176505309:82 , srv1077996948:83 , srv1136059698:83 , srv1310356261:83 , srv1624353758:83 , srv289480482:83 , srv544370942:83 , srv562273542:83 , srv109490929:84 , srv1194824963:84 , srv1283223431:84 , srv1320972383:84 , srv1377898458:84 , srv140461060:84 , srv1767334513:84 , srv178346533:84 , srv229815984:84 , srv268733977:84 , srv489158087:84 , srv681950842:84 , srv724117192:84 , srv741221188:84 , srv781843073:84 , srv1270354324:85 , srv1440779696:85 , srv1531853618:85 , srv1538325089:85 , srv1647051074:85 , srv1721279471:85 , srv1888826320:85 , srv1958339209:85 , srv2105149974:85 , srv2110360545:85 , srv332131412:85 , srv476619947:85 , srv558737949:85 , srv612231144:85 , srv618247639:85 , srv1024015885:86 , srv1265525689:86 , srv1412774548:86 , srv1588658112:86 , srv168744652:86 , srv254356802:86 , srv474768671:86 , srv669617599:86 , srv1177935051:87 , srv1478390128:87 , srv1531265960:87 , srv157744273:87 , srv1802588064:87 , srv182252191:87 , srv1847450513:87 , srv1866808472:87 , srv193985876:87 , srv405227456:87 , srv53057353:87 , srv580001053:87 , srv1147026089:88 , srv1195562931:88 , srv1466920157:88 , srv1580094725:88 , srv650192161:88 , srv735858498:88 , srv968523712:88 , srv1074622705:89 , srv13968576:89 , srv1567741296:89 , srv1954313090:89 , srv368109864:89 , srv394939:89 , srv556232993:89 , srv737623164:89 , srv738672848:89 , srv764034905:89 , srv973873837:89 , srv131782702:90 , srv1749632160:90 , srv2002498539:90 , srv865920488:90 , srv1420979409:91 , srv2017743610:91 , srv2090796402:91 , srv1891145529:92 , srv2028781987:92 , srv918556553:536 } 2024-02-28 01:53:18,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:18,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1412774548=21, srv2090235659=61, srv1538325089=30, srv157744273=33, srv1588658112=35, srv2105149974=63, srv1265525689=11, srv741221188=93, srv2078024885=60, srv544370942=79, srv168744652=38, srv193985876=53, srv1136059698=6, srv178346533=44, srv1767334513=43, srv1795110190=45, srv1866808472=50, srv558737949=81, srv2028781987=59, srv724117192=89, srv1624353758=36, srv1891145529=52, srv1575347272=32, srv1531265960=28, srv1077996948=3, srv368109864=71, srv1177935051=8, srv781843073=95, srv176505309=42, srv650192161=86, srv229815984=65, srv2110360545=64, srv1147026089=7, srv1074622705=2, srv1866416769=49, srv737623164=91, srv618247639=85, srv1888826320=51, srv145964263=25, srv254356802=66, srv562273542=82, srv764034905=94, srv556232993=80, srv109490929=4, srv53057353=78, srv1310356261=14, srv1420979409=22, srv140461060=20, srv681950842=88, srv1802588064=46, srv865920488=96, srv1435924811=23, srv268733977=67, srv474768671=74, srv1440779696=24, srv1466920157=26, srv405227456=73, srv1001017253=0, srv332131412=70, srv289480482=68, srv13968576=19, srv2090796402=62, srv1531853618=29, srv50652414=77, srv1958339209=55, srv1647051074=37, srv201701921=57, srv1195562931=10, srv2002498539=56, srv1749632160=41, srv580001053=83, srv1320972383=16, srv321699843=69, srv394939=72, srv973873837=99, srv1270354324=12, srv1954313090=54, srv1478390128=27, srv1024015885=1, srv182252191=47, srv1283223431=13, srv1580094725=34, srv1847450513=48, srv476619947=75, srv2017743610=58, srv612231144=84, srv1567741296=31, srv735858498=90, srv738672848=92, srv918556553=97, srv968523712=98, srv131782702=15, srv1377898458=18, srv1368066385=17, srv1721279471=39, srv489158087=76, srv1194824963=9, srv669617599=87, srv1745033871=40, srv1130993062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-28 01:53:18,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-28 01:53:18,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-28 01:53:18,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-28 01:53:18,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-28 01:53:18,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-28 01:53:18,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-28 01:53:18,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-28 01:53:18,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-28 01:53:18,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-28 01:53:18,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-28 01:53:18,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-28 01:53:18,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-28 01:53:18,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-28 01:53:19,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024442403751161725); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03801470564291989, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024442403751161725); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:19,008 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-28 01:53:19,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016546336074629894, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024442403751161725); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03801470564291989, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024442403751161725); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-28 01:53:19,288 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 395 ms to try 5000 different iterations. Found a solution that moves 1034 regions; Going from a computed imbalance of 0.016546336074629894 to a new imbalance of 7.747173177216663E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007444352311941399); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010865982916250669); MoveCostFunction : (multiplier=7.0, imbalance=0.11488888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007444352311941399); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:19,290 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1531265960:78 , srv1891145529:81 , srv1147026089:82 , srv1647051074:82 , srv2002498539:82 , srv2028781987:82 , srv865920488:82 , srv13968576:83 , srv1847450513:83 , srv476619947:83 , srv738672848:83 , srv1194824963:84 , srv1531853618:84 , srv1567741296:84 , srv1721279471:84 , srv1802588064:84 , srv1954313090:84 , srv394939:84 , srv580001053:84 , srv612231144:84 , srv724117192:84 , srv737623164:84 , srv764034905:84 , srv968523712:84 , srv1077996948:85 , srv109490929:85 , srv1265525689:85 , srv1320972383:85 , srv140461060:85 , srv1412774548:85 , srv1435924811:85 , srv1440779696:85 , srv145964263:85 , srv1538325089:85 , srv157744273:85 , srv1580094725:85 , srv1624353758:85 , srv1795110190:85 , srv182252191:85 , srv1866808472:85 , srv1958339209:85 , srv2017743610:85 , srv2078024885:85 , srv368109864:85 , srv489158087:85 , srv556232993:85 , srv618247639:85 , srv650192161:85 , srv735858498:85 , srv973873837:85 , srv1074622705:86 , srv1136059698:86 , srv1177935051:86 , srv1283223431:86 , srv1377898458:86 , srv1478390128:86 , srv1588658112:86 , srv1745033871:86 , srv1866416769:86 , srv1888826320:86 , srv193985876:86 , srv2105149974:86 , srv268733977:86 , srv332131412:86 , srv405227456:86 , srv544370942:86 , srv558737949:86 , srv562273542:86 , srv681950842:86 , srv741221188:86 , srv1130993062:87 , srv1195562931:87 , srv1270354324:87 , srv1420979409:87 , srv1749632160:87 , srv1767334513:87 , srv178346533:87 , srv2090796402:87 , srv2110360545:87 , srv229815984:87 , srv254356802:87 , srv289480482:87 , srv50652414:87 , srv53057353:87 , srv669617599:87 , srv1001017253:88 , srv131782702:88 , srv1368066385:88 , srv201701921:88 , srv321699843:88 , srv474768671:88 , srv781843073:88 , srv1024015885:89 , srv1310356261:89 , srv1466920157:89 , srv1575347272:89 , srv168744652:89 , srv176505309:89 , srv2090235659:89 , srv918556553:533 } 2024-02-28 01:53:19,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:19,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1412774548=21, srv2090235659=61, srv1538325089=30, srv157744273=33, srv1588658112=35, srv2105149974=63, srv1265525689=11, srv741221188=93, srv2078024885=60, srv544370942=79, srv168744652=38, srv193985876=53, srv1136059698=6, srv178346533=44, srv1767334513=43, srv1795110190=45, srv1866808472=50, srv558737949=81, srv2028781987=59, srv724117192=89, srv1624353758=36, srv1891145529=52, srv1575347272=32, srv1531265960=28, srv1077996948=3, srv368109864=71, srv1177935051=8, srv781843073=95, srv176505309=42, srv650192161=86, srv229815984=65, srv2110360545=64, srv1147026089=7, srv1074622705=2, srv1866416769=49, srv737623164=91, srv618247639=85, srv1888826320=51, srv145964263=25, srv254356802=66, srv562273542=82, srv764034905=94, srv556232993=80, srv109490929=4, srv53057353=78, srv1310356261=14, srv1420979409=22, srv140461060=20, srv681950842=88, srv1802588064=46, srv865920488=96, srv1435924811=23, srv268733977=67, srv474768671=74, srv1440779696=24, srv1466920157=26, srv405227456=73, srv1001017253=0, srv332131412=70, srv289480482=68, srv13968576=19, srv2090796402=62, srv1531853618=29, srv50652414=77, srv1958339209=55, srv1647051074=37, srv201701921=57, srv1195562931=10, srv2002498539=56, srv1749632160=41, srv580001053=83, srv1320972383=16, srv321699843=69, srv394939=72, srv973873837=99, srv1270354324=12, srv1954313090=54, srv1478390128=27, srv1024015885=1, srv182252191=47, srv1283223431=13, srv1580094725=34, srv1847450513=48, srv476619947=75, srv2017743610=58, srv612231144=84, srv1567741296=31, srv735858498=90, srv738672848=92, srv918556553=97, srv968523712=98, srv131782702=15, srv1377898458=18, srv1368066385=17, srv1721279471=39, srv489158087=76, srv1194824963=9, srv669617599=87, srv1745033871=40, srv1130993062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-28 01:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-28 01:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-28 01:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-28 01:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-28 01:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-28 01:53:19,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-28 01:53:19,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-28 01:53:19,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-28 01:53:19,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007444352311941399); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010865982916250669); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007444352311941399); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:19,429 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-28 01:53:19,430 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.674770949732825E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007444352311941399); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010865982916250669); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007444352311941399); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-28 01:53:19,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations. Found a solution that moves 275 regions; Going from a computed imbalance of 7.674770949732825E-4 to a new imbalance of 2.850820345631197E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.030555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:19,758 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1466920157:83 , srv157744273:83 , srv1767334513:83 , srv254356802:83 , srv1024015885:84 , srv1074622705:84 , srv109490929:84 , srv1147026089:84 , srv1177935051:84 , srv1194824963:84 , srv1195562931:84 , srv131782702:84 , srv1320972383:84 , srv1368066385:84 , srv1420979409:84 , srv1435924811:84 , srv145964263:84 , srv1531853618:84 , srv1538325089:84 , srv1580094725:84 , srv176505309:84 , srv1802588064:84 , srv1847450513:84 , srv1891145529:84 , srv2017743610:84 , srv2078024885:84 , srv2090235659:84 , srv2110360545:84 , srv229815984:84 , srv268733977:84 , srv289480482:84 , srv368109864:84 , srv544370942:84 , srv562273542:84 , srv618247639:84 , srv669617599:84 , srv738672848:84 , srv781843073:84 , srv973873837:84 , srv1001017253:85 , srv1130993062:85 , srv1265525689:85 , srv1270354324:85 , srv1283223431:85 , srv1310356261:85 , srv1377898458:85 , srv13968576:85 , srv140461060:85 , srv1412774548:85 , srv1478390128:85 , srv1531265960:85 , srv1567741296:85 , srv1575347272:85 , srv1588658112:85 , srv1624353758:85 , srv1647051074:85 , srv168744652:85 , srv1745033871:85 , srv1749632160:85 , srv178346533:85 , srv1795110190:85 , srv182252191:85 , srv1866416769:85 , srv1866808472:85 , srv1888826320:85 , srv193985876:85 , srv1954313090:85 , srv1958339209:85 , srv2002498539:85 , srv201701921:85 , srv2090796402:85 , srv2105149974:85 , srv321699843:85 , srv332131412:85 , srv394939:85 , srv405227456:85 , srv474768671:85 , srv489158087:85 , srv50652414:85 , srv53057353:85 , srv556232993:85 , srv580001053:85 , srv612231144:85 , srv681950842:85 , srv735858498:85 , srv737623164:85 , srv741221188:85 , srv865920488:85 , srv968523712:85 , srv1077996948:86 , srv1136059698:86 , srv1440779696:86 , srv1721279471:86 , srv2028781987:86 , srv476619947:86 , srv558737949:86 , srv650192161:86 , srv724117192:86 , srv764034905:86 , srv918556553:618 } 2024-02-28 01:53:19,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:19,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1412774548=21, srv2090235659=61, srv1538325089=30, srv157744273=33, srv1588658112=35, srv2105149974=63, srv1265525689=11, srv741221188=93, srv2078024885=60, srv544370942=79, srv168744652=38, srv193985876=53, srv1136059698=6, srv178346533=44, srv1767334513=43, srv1795110190=45, srv1866808472=50, srv558737949=81, srv2028781987=59, srv724117192=89, srv1624353758=36, srv1891145529=52, srv1575347272=32, srv1531265960=28, srv1077996948=3, srv368109864=71, srv1177935051=8, srv781843073=95, srv176505309=42, srv650192161=86, srv229815984=65, srv2110360545=64, srv1147026089=7, srv1074622705=2, srv1866416769=49, srv737623164=91, srv618247639=85, srv1888826320=51, srv145964263=25, srv254356802=66, srv562273542=82, srv764034905=94, srv556232993=80, srv109490929=4, srv53057353=78, srv1310356261=14, srv1420979409=22, srv140461060=20, srv681950842=88, srv1802588064=46, srv865920488=96, srv1435924811=23, srv268733977=67, srv474768671=74, srv1440779696=24, srv1466920157=26, srv405227456=73, srv1001017253=0, srv332131412=70, srv289480482=68, srv13968576=19, srv2090796402=62, srv1531853618=29, srv50652414=77, srv1958339209=55, srv1647051074=37, srv201701921=57, srv1195562931=10, srv2002498539=56, srv1749632160=41, srv580001053=83, srv1320972383=16, srv321699843=69, srv394939=72, srv973873837=99, srv1270354324=12, srv1954313090=54, srv1478390128=27, srv1024015885=1, srv182252191=47, srv1283223431=13, srv1580094725=34, srv1847450513=48, srv476619947=75, srv2017743610=58, srv612231144=84, srv1567741296=31, srv735858498=90, srv738672848=92, srv918556553=97, srv968523712=98, srv131782702=15, srv1377898458=18, srv1368066385=17, srv1721279471=39, srv489158087=76, srv1194824963=9, srv669617599=87, srv1745033871=40, srv1130993062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:19,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:19,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-28 01:53:19,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-28 01:53:19,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-28 01:53:19,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-28 01:53:19,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-28 01:53:19,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-28 01:53:19,894 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:19,895 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-28 01:53:19,895 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6582612299826927E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-28 01:53:20,119 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 27 regions; Going from a computed imbalance of 2.6582612299826927E-5 to a new imbalance of 7.57681554113109E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:20,120 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1077996948:83 , srv1377898458:83 , srv1575347272:83 , srv1721279471:83 , srv474768671:83 , srv53057353:83 , srv737623164:83 , srv764034905:83 , srv968523712:83 , srv1001017253:84 , srv1024015885:84 , srv1074622705:84 , srv109490929:84 , srv1130993062:84 , srv1136059698:84 , srv1147026089:84 , srv1177935051:84 , srv1194824963:84 , srv1195562931:84 , srv1265525689:84 , srv1270354324:84 , srv1283223431:84 , srv1310356261:84 , srv131782702:84 , srv1368066385:84 , srv13968576:84 , srv140461060:84 , srv1412774548:84 , srv1420979409:84 , srv1435924811:84 , srv1440779696:84 , srv1466920157:84 , srv1478390128:84 , srv1531265960:84 , srv1538325089:84 , srv1567741296:84 , srv1588658112:84 , srv1624353758:84 , srv1647051074:84 , srv168744652:84 , srv1745033871:84 , srv1749632160:84 , srv176505309:84 , srv1767334513:84 , srv178346533:84 , srv1802588064:84 , srv182252191:84 , srv1847450513:84 , srv1866416769:84 , srv1866808472:84 , srv1888826320:84 , srv1891145529:84 , srv193985876:84 , srv1954313090:84 , srv1958339209:84 , srv2002498539:84 , srv201701921:84 , srv2017743610:84 , srv2028781987:84 , srv2078024885:84 , srv2090235659:84 , srv2090796402:84 , srv2105149974:84 , srv2110360545:84 , srv229815984:84 , srv254356802:84 , srv268733977:84 , srv321699843:84 , srv332131412:84 , srv394939:84 , srv405227456:84 , srv476619947:84 , srv489158087:84 , srv50652414:84 , srv544370942:84 , srv556232993:84 , srv558737949:84 , srv562273542:84 , srv580001053:84 , srv612231144:84 , srv650192161:84 , srv669617599:84 , srv681950842:84 , srv724117192:84 , srv735858498:84 , srv738672848:84 , srv741221188:84 , srv781843073:84 , srv865920488:84 , srv973873837:84 , srv1320972383:85 , srv145964263:85 , srv1531853618:85 , srv157744273:85 , srv1580094725:85 , srv1795110190:85 , srv289480482:85 , srv368109864:85 , srv618247639:85 , srv918556553:684 } 2024-02-28 01:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1412774548=21, srv2090235659=61, srv1538325089=30, srv157744273=33, srv1588658112=35, srv2105149974=63, srv1265525689=11, srv741221188=93, srv2078024885=60, srv544370942=79, srv168744652=38, srv193985876=53, srv1136059698=6, srv178346533=44, srv1767334513=43, srv1795110190=45, srv1866808472=50, srv558737949=81, srv2028781987=59, srv724117192=89, srv1624353758=36, srv1891145529=52, srv1575347272=32, srv1531265960=28, srv1077996948=3, srv368109864=71, srv1177935051=8, srv781843073=95, srv176505309=42, srv650192161=86, srv229815984=65, srv2110360545=64, srv1147026089=7, srv1074622705=2, srv1866416769=49, srv737623164=91, srv618247639=85, srv1888826320=51, srv145964263=25, srv254356802=66, srv562273542=82, srv764034905=94, srv556232993=80, srv109490929=4, srv53057353=78, srv1310356261=14, srv1420979409=22, srv140461060=20, srv681950842=88, srv1802588064=46, srv865920488=96, srv1435924811=23, srv268733977=67, srv474768671=74, srv1440779696=24, srv1466920157=26, srv405227456=73, srv1001017253=0, srv332131412=70, srv289480482=68, srv13968576=19, srv2090796402=62, srv1531853618=29, srv50652414=77, srv1958339209=55, srv1647051074=37, srv201701921=57, srv1195562931=10, srv2002498539=56, srv1749632160=41, srv580001053=83, srv1320972383=16, srv321699843=69, srv394939=72, srv973873837=99, srv1270354324=12, srv1954313090=54, srv1478390128=27, srv1024015885=1, srv182252191=47, srv1283223431=13, srv1580094725=34, srv1847450513=48, srv476619947=75, srv2017743610=58, srv612231144=84, srv1567741296=31, srv735858498=90, srv738672848=92, srv918556553=97, srv968523712=98, srv131782702=15, srv1377898458=18, srv1368066385=17, srv1721279471=39, srv489158087=76, srv1194824963=9, srv669617599=87, srv1745033871=40, srv1130993062=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:20,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:20,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:20,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:20,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:20,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:20,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-28 01:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-28 01:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-28 01:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-28 01:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-28 01:53:20,248 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:20,248 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1077996948:83 , srv1377898458:83 , srv1575347272:83 , srv1721279471:83 , srv474768671:83 , srv53057353:83 , srv737623164:83 , srv764034905:83 , srv968523712:83 , srv1001017253:84 , srv1024015885:84 , srv1074622705:84 , srv109490929:84 , srv1130993062:84 , srv1136059698:84 , srv1147026089:84 , srv1177935051:84 , srv1194824963:84 , srv1195562931:84 , srv1265525689:84 , srv1270354324:84 , srv1283223431:84 , srv1310356261:84 , srv131782702:84 , srv1368066385:84 , srv13968576:84 , srv140461060:84 , srv1412774548:84 , srv1420979409:84 , srv1435924811:84 , srv1440779696:84 , srv1466920157:84 , srv1478390128:84 , srv1531265960:84 , srv1538325089:84 , srv1567741296:84 , srv1588658112:84 , srv1624353758:84 , srv1647051074:84 , srv168744652:84 , srv1745033871:84 , srv1749632160:84 , srv176505309:84 , srv1767334513:84 , srv178346533:84 , srv1802588064:84 , srv182252191:84 , srv1847450513:84 , srv1866416769:84 , srv1866808472:84 , srv1888826320:84 , srv1891145529:84 , srv193985876:84 , srv1954313090:84 , srv1958339209:84 , srv2002498539:84 , srv201701921:84 , srv2017743610:84 , srv2028781987:84 , srv2078024885:84 , srv2090235659:84 , srv2090796402:84 , srv2105149974:84 , srv2110360545:84 , srv229815984:84 , srv254356802:84 , srv268733977:84 , srv321699843:84 , srv332131412:84 , srv394939:84 , srv405227456:84 , srv476619947:84 , srv489158087:84 , srv50652414:84 , srv544370942:84 , srv556232993:84 , srv558737949:84 , srv562273542:84 , srv580001053:84 , srv612231144:84 , srv650192161:84 , srv669617599:84 , srv681950842:84 , srv724117192:84 , srv735858498:84 , srv738672848:84 , srv741221188:84 , srv781843073:84 , srv865920488:84 , srv973873837:84 , srv1320972383:85 , srv145964263:85 , srv1531853618:85 , srv157744273:85 , srv1580094725:85 , srv1795110190:85 , srv289480482:85 , srv368109864:85 , srv618247639:85 , srv918556553:684 } 2024-02-28 01:53:20,273 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=285 (was 285), ProcessCount=172 (was 172), AvailableMemoryMB=10593 (was 11331) 2024-02-28 01:53:20,282 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=172, AvailableMemoryMB=10592 2024-02-28 01:53:20,283 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-28 01:53:20,283 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-28 01:53:20,290 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023523291:84 , srv1038316851:84 , srv1291026350:84 , srv1306458716:84 , srv1396051598:84 , srv1413590483:84 , srv1431705927:84 , srv1438976974:84 , srv1451646653:84 , srv1550538083:84 , srv1605194109:84 , srv1609537326:84 , srv1772518486:84 , srv1882223221:84 , srv1961548982:84 , srv1967113904:84 , srv20443665:84 , srv22800662:84 , srv330996234:84 , srv375065897:84 , srv525853688:84 , srv559065696:84 , srv583387767:84 , srv698164105:84 , srv716588438:84 , srv770675850:84 , srv935948088:84 , srv982271936:84 , srv997331484:84 , srv6885548:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-28 01:53:20,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:53:20,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv559065696=21, srv20443665=16, srv1438976974=7, srv22800662=17, srv982271936=28, srv1023523291=0, srv6885548=23, srv1396051598=4, srv1291026350=2, srv1961548982=14, srv1605194109=10, srv1609537326=11, srv1306458716=3, srv330996234=18, srv525853688=20, srv583387767=22, srv1967113904=15, srv1038316851=1, srv1413590483=5, srv1431705927=6, srv375065897=19, srv935948088=27, srv1451646653=8, srv1772518486=12, srv716588438=25, srv698164105=24, srv1550538083=9, srv1882223221=13, srv770675850=26, srv997331484=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:53:20,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-28 01:53:20,316 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1258882700338486, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:53:20,316 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912589713092526, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1258882700338486, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-28 01:55:20,304 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6158439 different iterations. Found a solution that moves 1842 regions; Going from a computed imbalance of 0.9912589713092526 to a new imbalance of 4.2993198912066E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:55:20,310 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1023523291:90 , srv1038316851:90 , srv1291026350:90 , srv1306458716:90 , srv1396051598:90 , srv1413590483:90 , srv1431705927:90 , srv1438976974:90 , srv1451646653:90 , srv1550538083:90 , srv1605194109:90 , srv1609537326:90 , srv1772518486:90 , srv1882223221:90 , srv1961548982:90 , srv1967113904:90 , srv20443665:90 , srv22800662:90 , srv330996234:90 , srv375065897:90 , srv525853688:90 , srv559065696:90 , srv583387767:90 , srv6885548:90 , srv698164105:90 , srv716588438:90 , srv770675850:90 , srv935948088:90 , srv982271936:90 , srv997331484:90 } 2024-02-28 01:55:20,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-28 01:55:20,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv559065696=21, srv20443665=16, srv1438976974=7, srv22800662=17, srv982271936=28, srv1023523291=0, srv6885548=23, srv1396051598=4, srv1291026350=2, srv1961548982=14, srv1605194109=10, srv1609537326=11, srv1306458716=3, srv330996234=18, srv525853688=20, srv583387767=22, srv1967113904=15, srv1038316851=1, srv1413590483=5, srv1431705927=6, srv375065897=19, srv935948088=27, srv1451646653=8, srv1772518486=12, srv716588438=25, srv698164105=24, srv1550538083=9, srv1882223221=13, srv770675850=26, srv997331484=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-28 01:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-28 01:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-28 01:55:20,328 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-28 01:55:20,328 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1023523291:90 , srv1038316851:90 , srv1291026350:90 , srv1306458716:90 , srv1396051598:90 , srv1413590483:90 , srv1431705927:90 , srv1438976974:90 , srv1451646653:90 , srv1550538083:90 , srv1605194109:90 , srv1609537326:90 , srv1772518486:90 , srv1882223221:90 , srv1961548982:90 , srv1967113904:90 , srv20443665:90 , srv22800662:90 , srv330996234:90 , srv375065897:90 , srv525853688:90 , srv559065696:90 , srv583387767:90 , srv6885548:90 , srv698164105:90 , srv716588438:90 , srv770675850:90 , srv935948088:90 , srv982271936:90 , srv997331484:90 } 2024-02-28 01:55:20,342 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=211 (was 285), ProcessCount=171 (was 172), AvailableMemoryMB=10175 (was 10592)