2023-05-03 23:56:45,319 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-03 23:56:45,569 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-03 23:56:45,660 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-03 23:56:45,708 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-03 23:56:45,714 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-03 23:56:45,724 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=75, ProcessCount=167, AvailableMemoryMB=8349 2023-05-03 23:56:45,726 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-03 23:56:45,726 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-03 23:56:45,829 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013784908:84 , srv1036070409:84 , srv1044047461:84 , srv1064332542:84 , srv1071823776:84 , srv1083374500:84 , srv1102197047:84 , srv1111917198:84 , srv114438135:84 , srv1159512086:84 , srv1180773690:84 , srv1216945804:84 , srv125782310:84 , srv1275217638:84 , srv1275735693:84 , srv1287100506:84 , srv1289094034:84 , srv1293883056:84 , srv1319222276:84 , srv1345072397:84 , srv1371386428:84 , srv1411917757:84 , srv144158940:84 , srv1534283550:84 , srv1539590779:84 , srv1564967788:84 , srv1586135533:84 , srv1586664071:84 , srv1610358875:84 , srv1613204003:84 , srv1706710258:84 , srv1721103766:84 , srv1730914430:84 , srv174233906:84 , srv1743862113:84 , srv175829666:84 , srv1780001545:84 , srv1783820125:84 , srv1791647067:84 , srv1798604912:84 , srv1851324240:84 , srv1867450694:84 , srv1873148050:84 , srv1873500966:84 , srv1892046993:84 , srv192144871:84 , srv1939139636:84 , srv1942509372:84 , srv1975811962:84 , srv198186173:84 , srv2001918669:84 , srv2019001641:84 , srv2033576866:84 , srv2088377321:84 , srv2089218254:84 , srv2095361519:84 , srv2104040000:84 , srv2110911632:84 , srv2115889863:84 , srv2128011265:84 , srv220634207:84 , srv22499753:84 , srv232976865:84 , srv233436421:84 , srv243049872:84 , srv259035520:84 , srv268539222:84 , srv315288037:84 , srv325479144:84 , srv346829946:84 , srv423404518:84 , srv431921144:84 , srv436737872:84 , srv43846102:84 , srv448202216:84 , srv466786590:84 , srv467385624:84 , srv483071537:84 , srv488053818:84 , srv529692366:84 , srv558060772:84 , srv588828339:84 , srv603161703:84 , srv736495810:84 , srv764174477:84 , srv774719147:84 , srv798949527:84 , srv799065527:84 , srv801028291:84 , srv833582759:84 , srv837043795:84 , srv860866279:84 , srv864457299:84 , srv886601191:84 , srv90047612:84 , srv913278986:84 , srv927728843:84 , srv932855811:84 , srv967514316:84 , srv671057643:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-03 23:56:45,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:45,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv488053818=78, srv175829666=35, srv1534283550=23, srv268539222=66, srv1586135533=26, srv1975811962=48, srv588828339=81, srv114438135=8, srv1873500966=43, srv436737872=72, srv1275217638=13, srv174233906=33, srv220634207=60, srv798949527=87, srv1180773690=10, srv325479144=68, srv1867450694=41, srv1564967788=25, srv466786590=75, srv192144871=45, srv43846102=73, srv423404518=70, srv558060772=80, srv860866279=92, srv467385624=76, srv1293883056=17, srv90047612=95, srv1083374500=5, srv1345072397=19, srv144158940=22, srv1610358875=28, srv1586664071=27, srv1873148050=42, srv232976865=62, srv774719147=86, srv967514316=99, srv431921144=71, srv1613204003=29, srv1287100506=15, srv529692366=79, srv346829946=69, srv927728843=97, srv1743862113=34, srv1539590779=24, srv483071537=77, srv913278986=96, srv2019001641=51, srv1706710258=30, srv801028291=89, srv1319222276=18, srv1942509372=47, srv2128011265=59, srv603161703=82, srv1064332542=3, srv833582759=90, srv315288037=67, srv1411917757=21, srv1289094034=16, srv1102197047=6, srv1216945804=11, srv22499753=61, srv198186173=49, srv2110911632=57, srv1721103766=31, srv1275735693=14, srv1044047461=2, srv2033576866=52, srv2088377321=53, srv1371386428=20, srv1851324240=40, srv259035520=65, srv837043795=91, srv2095361519=55, srv1036070409=1, srv799065527=88, srv1892046993=44, srv886601191=94, srv1783820125=37, srv2089218254=54, srv448202216=74, srv1111917198=7, srv233436421=63, srv1780001545=36, srv243049872=64, srv671057643=83, srv1730914430=32, srv2104040000=56, srv764174477=85, srv125782310=12, srv1798604912=39, srv736495810=84, srv2115889863=58, srv864457299=93, srv1013784908=0, srv932855811=98, srv2001918669=50, srv1071823776=4, srv1939139636=46, srv1159512086=9, srv1791647067=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:45,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:45,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:45,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:45,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:45,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:45,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:45,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-03 23:56:45,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-03 23:56:45,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-03 23:56:45,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-03 23:56:45,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-03 23:56:45,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-03 23:56:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-03 23:56:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-03 23:56:46,151 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:46,151 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-03 23:56:46,151 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-03 23:56:46,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations. Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17990713593160257. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039366159015331956, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05434230782505548, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039366159015331956, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17533333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:46,422 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv232976865:82 , srv259035520:82 , srv529692366:82 , srv1013784908:83 , srv1083374500:83 , srv1102197047:83 , srv1216945804:83 , srv1783820125:83 , srv243049872:83 , srv603161703:83 , srv764174477:83 , srv125782310:84 , srv1411917757:84 , srv175829666:84 , srv22499753:84 , srv325479144:84 , srv558060772:84 , srv1036070409:85 , srv114438135:85 , srv1275735693:85 , srv1319222276:85 , srv1371386428:85 , srv1791647067:85 , srv1873148050:85 , srv1873500966:85 , srv1892046993:85 , srv233436421:85 , srv268539222:85 , srv43846102:85 , srv467385624:85 , srv488053818:85 , srv798949527:85 , srv837043795:85 , srv860866279:85 , srv967514316:85 , srv1064332542:86 , srv1534283550:86 , srv1610358875:86 , srv1721103766:86 , srv1780001545:86 , srv1867450694:86 , srv1939139636:86 , srv1975811962:86 , srv2089218254:86 , srv2104040000:86 , srv448202216:86 , srv799065527:86 , srv833582759:86 , srv886601191:86 , srv90047612:86 , srv927728843:86 , srv1044047461:87 , srv1159512086:87 , srv144158940:87 , srv1539590779:87 , srv1564967788:87 , srv1586135533:87 , srv1613204003:87 , srv1706710258:87 , srv174233906:87 , srv1743862113:87 , srv192144871:87 , srv1942509372:87 , srv588828339:87 , srv736495810:87 , srv774719147:87 , srv864457299:87 , srv932855811:87 , srv1111917198:88 , srv1180773690:88 , srv1287100506:88 , srv1289094034:88 , srv1293883056:88 , srv1345072397:88 , srv1586664071:88 , srv1798604912:88 , srv198186173:88 , srv2033576866:88 , srv315288037:88 , srv423404518:88 , srv466786590:88 , srv483071537:88 , srv801028291:88 , srv1275217638:89 , srv1730914430:89 , srv1851324240:89 , srv2019001641:89 , srv2095361519:89 , srv2110911632:89 , srv2128011265:89 , srv346829946:89 , srv913278986:89 , srv2001918669:90 , srv431921144:90 , srv436737872:90 , srv2088377321:91 , srv2115889863:91 , srv220634207:92 , srv1071823776:96 , srv671057643:440 } 2023-05-03 23:56:46,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:46,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv488053818=78, srv175829666=35, srv1534283550=23, srv268539222=66, srv1586135533=26, srv1975811962=48, srv588828339=81, srv114438135=8, srv1873500966=43, srv436737872=72, srv1275217638=13, srv174233906=33, srv220634207=60, srv798949527=87, srv1180773690=10, srv325479144=68, srv1867450694=41, srv1564967788=25, srv466786590=75, srv192144871=45, srv43846102=73, srv423404518=70, srv558060772=80, srv860866279=92, srv467385624=76, srv1293883056=17, srv90047612=95, srv1083374500=5, srv1345072397=19, srv144158940=22, srv1610358875=28, srv1586664071=27, srv1873148050=42, srv232976865=62, srv774719147=86, srv967514316=99, srv431921144=71, srv1613204003=29, srv1287100506=15, srv529692366=79, srv346829946=69, srv927728843=97, srv1743862113=34, srv1539590779=24, srv483071537=77, srv913278986=96, srv2019001641=51, srv1706710258=30, srv801028291=89, srv1319222276=18, srv1942509372=47, srv2128011265=59, srv603161703=82, srv1064332542=3, srv833582759=90, srv315288037=67, srv1411917757=21, srv1289094034=16, srv1102197047=6, srv1216945804=11, srv22499753=61, srv198186173=49, srv2110911632=57, srv1721103766=31, srv1275735693=14, srv1044047461=2, srv2033576866=52, srv2088377321=53, srv1371386428=20, srv1851324240=40, srv259035520=65, srv837043795=91, srv2095361519=55, srv1036070409=1, srv799065527=88, srv1892046993=44, srv886601191=94, srv1783820125=37, srv2089218254=54, srv448202216=74, srv1111917198=7, srv233436421=63, srv1780001545=36, srv243049872=64, srv671057643=83, srv1730914430=32, srv2104040000=56, srv764174477=85, srv125782310=12, srv1798604912=39, srv736495810=84, srv2115889863=58, srv864457299=93, srv1013784908=0, srv932855811=98, srv2001918669=50, srv1071823776=4, srv1939139636=46, srv1159512086=9, srv1791647067=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:46,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:46,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:46,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:46,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:46,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:46,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:46,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-03 23:56:46,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-03 23:56:46,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-03 23:56:46,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-03 23:56:46,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-03 23:56:46,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-03 23:56:46,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-03 23:56:46,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-03 23:56:46,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-03 23:56:46,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-03 23:56:46,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-03 23:56:46,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-03 23:56:46,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-03 23:56:46,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-03 23:56:46,616 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039366159015331956, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05434230782505548, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039366159015331956, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17533333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:46,616 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-03 23:56:46,616 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17987430985399677, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039366159015331956, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05434230782505548, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039366159015331956, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17533333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-03 23:56:46,813 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 321 ms to try 5000 different iterations. Found a solution that moves 2675 regions; Going from a computed imbalance of 0.17987430985399677 to a new imbalance of 0.014890398859822039. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020155277011749965); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03223441079122184, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2972222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020155277011749965); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:46,826 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1071823776:78 , srv2115889863:80 , srv1275217638:81 , srv2088377321:81 , srv431921144:81 , srv1730914430:82 , srv2001918669:82 , srv2019001641:82 , srv2095361519:82 , srv220634207:82 , srv483071537:82 , srv1111917198:83 , srv2128011265:83 , srv315288037:83 , srv346829946:83 , srv423404518:83 , srv436737872:83 , srv736495810:83 , srv1044047461:84 , srv1180773690:84 , srv1539590779:84 , srv1798604912:84 , srv192144871:84 , srv198186173:84 , srv2033576866:84 , srv2110911632:84 , srv860866279:84 , srv1159512086:85 , srv1289094034:85 , srv1293883056:85 , srv1319222276:85 , srv1345072397:85 , srv144158940:85 , srv1534283550:85 , srv1586135533:85 , srv1706710258:85 , srv1721103766:85 , srv1743862113:85 , srv1780001545:85 , srv1942509372:85 , srv448202216:85 , srv774719147:85 , srv833582759:85 , srv913278986:85 , srv1036070409:86 , srv1064332542:86 , srv1083374500:86 , srv1586664071:86 , srv1610358875:86 , srv1851324240:86 , srv1867450694:86 , srv1873148050:86 , srv1975811962:86 , srv2089218254:86 , srv233436421:86 , srv268539222:86 , srv466786590:86 , srv798949527:86 , srv799065527:86 , srv801028291:86 , srv90047612:86 , srv114438135:87 , srv1216945804:87 , srv1275735693:87 , srv1287100506:87 , srv1371386428:87 , srv1411917757:87 , srv1564967788:87 , srv325479144:87 , srv43846102:87 , srv588828339:87 , srv864457299:87 , srv886601191:87 , srv927728843:87 , srv932855811:87 , srv967514316:87 , srv1102197047:88 , srv125782310:88 , srv175829666:88 , srv1791647067:88 , srv1873500966:88 , srv259035520:88 , srv467385624:88 , srv488053818:88 , srv837043795:88 , srv1013784908:89 , srv174233906:89 , srv1892046993:89 , srv2104040000:89 , srv22499753:89 , srv243049872:89 , srv529692366:89 , srv603161703:89 , srv1613204003:90 , srv1783820125:90 , srv1939139636:90 , srv558060772:90 , srv764174477:90 , srv232976865:91 , srv671057643:513 } 2023-05-03 23:56:46,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:46,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv488053818=78, srv175829666=35, srv1534283550=23, srv268539222=66, srv1586135533=26, srv1975811962=48, srv588828339=81, srv114438135=8, srv1873500966=43, srv436737872=72, srv1275217638=13, srv174233906=33, srv220634207=60, srv798949527=87, srv1180773690=10, srv325479144=68, srv1867450694=41, srv1564967788=25, srv466786590=75, srv192144871=45, srv43846102=73, srv423404518=70, srv558060772=80, srv860866279=92, srv467385624=76, srv1293883056=17, srv90047612=95, srv1083374500=5, srv1345072397=19, srv144158940=22, srv1610358875=28, srv1586664071=27, srv1873148050=42, srv232976865=62, srv774719147=86, srv967514316=99, srv431921144=71, srv1613204003=29, srv1287100506=15, srv529692366=79, srv346829946=69, srv927728843=97, srv1743862113=34, srv1539590779=24, srv483071537=77, srv913278986=96, srv2019001641=51, srv1706710258=30, srv801028291=89, srv1319222276=18, srv1942509372=47, srv2128011265=59, srv603161703=82, srv1064332542=3, srv833582759=90, srv315288037=67, srv1411917757=21, srv1289094034=16, srv1102197047=6, srv1216945804=11, srv22499753=61, srv198186173=49, srv2110911632=57, srv1721103766=31, srv1275735693=14, srv1044047461=2, srv2033576866=52, srv2088377321=53, srv1371386428=20, srv1851324240=40, srv259035520=65, srv837043795=91, srv2095361519=55, srv1036070409=1, srv799065527=88, srv1892046993=44, srv886601191=94, srv1783820125=37, srv2089218254=54, srv448202216=74, srv1111917198=7, srv233436421=63, srv1780001545=36, srv243049872=64, srv671057643=83, srv1730914430=32, srv2104040000=56, srv764174477=85, srv125782310=12, srv1798604912=39, srv736495810=84, srv2115889863=58, srv864457299=93, srv1013784908=0, srv932855811=98, srv2001918669=50, srv1071823776=4, srv1939139636=46, srv1159512086=9, srv1791647067=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:46,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:46,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:46,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:46,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:46,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:46,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-03 23:56:46,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-03 23:56:46,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-03 23:56:46,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-03 23:56:46,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-03 23:56:46,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-03 23:56:46,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-03 23:56:46,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-03 23:56:46,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-03 23:56:46,956 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020155277011749965); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03223441079122184, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020155277011749965); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:46,957 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-03 23:56:46,957 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014871668109481685, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020155277011749965); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03223441079122184, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020155277011749965); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-03 23:56:47,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 232 ms to try 5000 different iterations. Found a solution that moves 1005 regions; Going from a computed imbalance of 0.014871668109481685 to a new imbalance of 7.610542007048846E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0066647958467430145); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00704323668071124); MoveCostFunction : (multiplier=7.0, imbalance=0.11166666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0066647958467430145); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,118 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1613204003:81 , srv174233906:82 , srv1939139636:82 , srv220634207:82 , srv1287100506:83 , srv1293883056:83 , srv1586664071:83 , srv2104040000:83 , srv864457299:83 , srv927728843:83 , srv1064332542:84 , srv1071823776:84 , srv1289094034:84 , srv1345072397:84 , srv1564967788:84 , srv1783820125:84 , srv1791647067:84 , srv1851324240:84 , srv1873500966:84 , srv1892046993:84 , srv466786590:84 , srv467385624:84 , srv488053818:84 , srv558060772:84 , srv588828339:84 , srv764174477:84 , srv801028291:84 , srv837043795:84 , srv913278986:84 , srv932855811:84 , srv1111917198:85 , srv125782310:85 , srv1275735693:85 , srv144158940:85 , srv1586135533:85 , srv1610358875:85 , srv1706710258:85 , srv175829666:85 , srv1798604912:85 , srv1867450694:85 , srv1975811962:85 , srv2001918669:85 , srv2033576866:85 , srv2110911632:85 , srv22499753:85 , srv232976865:85 , srv233436421:85 , srv243049872:85 , srv436737872:85 , srv43846102:85 , srv529692366:85 , srv603161703:85 , srv774719147:85 , srv886601191:85 , srv90047612:85 , srv967514316:85 , srv1013784908:86 , srv114438135:86 , srv1159512086:86 , srv1180773690:86 , srv1216945804:86 , srv1534283550:86 , srv1539590779:86 , srv1721103766:86 , srv1730914430:86 , srv1743862113:86 , srv1780001545:86 , srv192144871:86 , srv1942509372:86 , srv198186173:86 , srv2019001641:86 , srv2088377321:86 , srv2089218254:86 , srv2128011265:86 , srv268539222:86 , srv315288037:86 , srv325479144:86 , srv346829946:86 , srv423404518:86 , srv448202216:86 , srv798949527:86 , srv799065527:86 , srv1036070409:87 , srv1044047461:87 , srv1102197047:87 , srv1319222276:87 , srv1371386428:87 , srv1411917757:87 , srv1873148050:87 , srv2095361519:87 , srv259035520:87 , srv431921144:87 , srv833582759:87 , srv1083374500:88 , srv1275217638:88 , srv2115889863:88 , srv483071537:88 , srv736495810:88 , srv860866279:88 , srv671057643:564 } 2023-05-03 23:56:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv488053818=78, srv175829666=35, srv1534283550=23, srv268539222=66, srv1586135533=26, srv1975811962=48, srv588828339=81, srv114438135=8, srv1873500966=43, srv436737872=72, srv1275217638=13, srv174233906=33, srv220634207=60, srv798949527=87, srv1180773690=10, srv325479144=68, srv1867450694=41, srv1564967788=25, srv466786590=75, srv192144871=45, srv43846102=73, srv423404518=70, srv558060772=80, srv860866279=92, srv467385624=76, srv1293883056=17, srv90047612=95, srv1083374500=5, srv1345072397=19, srv144158940=22, srv1610358875=28, srv1586664071=27, srv1873148050=42, srv232976865=62, srv774719147=86, srv967514316=99, srv431921144=71, srv1613204003=29, srv1287100506=15, srv529692366=79, srv346829946=69, srv927728843=97, srv1743862113=34, srv1539590779=24, srv483071537=77, srv913278986=96, srv2019001641=51, srv1706710258=30, srv801028291=89, srv1319222276=18, srv1942509372=47, srv2128011265=59, srv603161703=82, srv1064332542=3, srv833582759=90, srv315288037=67, srv1411917757=21, srv1289094034=16, srv1102197047=6, srv1216945804=11, srv22499753=61, srv198186173=49, srv2110911632=57, srv1721103766=31, srv1275735693=14, srv1044047461=2, srv2033576866=52, srv2088377321=53, srv1371386428=20, srv1851324240=40, srv259035520=65, srv837043795=91, srv2095361519=55, srv1036070409=1, srv799065527=88, srv1892046993=44, srv886601191=94, srv1783820125=37, srv2089218254=54, srv448202216=74, srv1111917198=7, srv233436421=63, srv1780001545=36, srv243049872=64, srv671057643=83, srv1730914430=32, srv2104040000=56, srv764174477=85, srv125782310=12, srv1798604912=39, srv736495810=84, srv2115889863=58, srv864457299=93, srv1013784908=0, srv932855811=98, srv2001918669=50, srv1071823776=4, srv1939139636=46, srv1159512086=9, srv1791647067=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:47,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:47,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-03 23:56:47,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-03 23:56:47,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-03 23:56:47,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-03 23:56:47,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-03 23:56:47,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-03 23:56:47,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-03 23:56:47,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-03 23:56:47,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-03 23:56:47,217 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0066647958467430145); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00704323668071124); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0066647958467430145); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,217 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-03 23:56:47,218 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.540170402966392E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0066647958467430145); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00704323668071124); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0066647958467430145); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-03 23:56:47,434 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations. Found a solution that moves 265 regions; Going from a computed imbalance of 7.540170402966392E-4 to a new imbalance of 3.590784982628643E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.029444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,435 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1371386428:83 , srv1851324240:83 , srv2115889863:83 , srv1013784908:84 , srv1036070409:84 , srv1044047461:84 , srv1071823776:84 , srv1102197047:84 , srv114438135:84 , srv1159512086:84 , srv1180773690:84 , srv1275217638:84 , srv1287100506:84 , srv1411917757:84 , srv1564967788:84 , srv1613204003:84 , srv174233906:84 , srv1743862113:84 , srv1873148050:84 , srv1892046993:84 , srv1939139636:84 , srv1942509372:84 , srv198186173:84 , srv2088377321:84 , srv2089218254:84 , srv2095361519:84 , srv2104040000:84 , srv2110911632:84 , srv2128011265:84 , srv22499753:84 , srv232976865:84 , srv346829946:84 , srv431921144:84 , srv466786590:84 , srv483071537:84 , srv558060772:84 , srv588828339:84 , srv736495810:84 , srv801028291:84 , srv833582759:84 , srv837043795:84 , srv886601191:84 , srv932855811:84 , srv1083374500:85 , srv1111917198:85 , srv125782310:85 , srv1275735693:85 , srv1289094034:85 , srv1319222276:85 , srv1345072397:85 , srv144158940:85 , srv1534283550:85 , srv1539590779:85 , srv1586135533:85 , srv1586664071:85 , srv1610358875:85 , srv1706710258:85 , srv1721103766:85 , srv1730914430:85 , srv175829666:85 , srv1780001545:85 , srv1783820125:85 , srv1791647067:85 , srv1798604912:85 , srv1867450694:85 , srv1873500966:85 , srv192144871:85 , srv1975811962:85 , srv2001918669:85 , srv2019001641:85 , srv2033576866:85 , srv243049872:85 , srv259035520:85 , srv268539222:85 , srv315288037:85 , srv325479144:85 , srv423404518:85 , srv436737872:85 , srv43846102:85 , srv448202216:85 , srv467385624:85 , srv488053818:85 , srv529692366:85 , srv603161703:85 , srv764174477:85 , srv774719147:85 , srv798949527:85 , srv799065527:85 , srv860866279:85 , srv864457299:85 , srv90047612:85 , srv913278986:85 , srv967514316:85 , srv1064332542:86 , srv1216945804:86 , srv1293883056:86 , srv220634207:86 , srv233436421:86 , srv927728843:86 , srv671057643:625 } 2023-05-03 23:56:47,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:47,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv488053818=78, srv175829666=35, srv1534283550=23, srv268539222=66, srv1586135533=26, srv1975811962=48, srv588828339=81, srv114438135=8, srv1873500966=43, srv436737872=72, srv1275217638=13, srv174233906=33, srv220634207=60, srv798949527=87, srv1180773690=10, srv325479144=68, srv1867450694=41, srv1564967788=25, srv466786590=75, srv192144871=45, srv43846102=73, srv423404518=70, srv558060772=80, srv860866279=92, srv467385624=76, srv1293883056=17, srv90047612=95, srv1083374500=5, srv1345072397=19, srv144158940=22, srv1610358875=28, srv1586664071=27, srv1873148050=42, srv232976865=62, srv774719147=86, srv967514316=99, srv431921144=71, srv1613204003=29, srv1287100506=15, srv529692366=79, srv346829946=69, srv927728843=97, srv1743862113=34, srv1539590779=24, srv483071537=77, srv913278986=96, srv2019001641=51, srv1706710258=30, srv801028291=89, srv1319222276=18, srv1942509372=47, srv2128011265=59, srv603161703=82, srv1064332542=3, srv833582759=90, srv315288037=67, srv1411917757=21, srv1289094034=16, srv1102197047=6, srv1216945804=11, srv22499753=61, srv198186173=49, srv2110911632=57, srv1721103766=31, srv1275735693=14, srv1044047461=2, srv2033576866=52, srv2088377321=53, srv1371386428=20, srv1851324240=40, srv259035520=65, srv837043795=91, srv2095361519=55, srv1036070409=1, srv799065527=88, srv1892046993=44, srv886601191=94, srv1783820125=37, srv2089218254=54, srv448202216=74, srv1111917198=7, srv233436421=63, srv1780001545=36, srv243049872=64, srv671057643=83, srv1730914430=32, srv2104040000=56, srv764174477=85, srv125782310=12, srv1798604912=39, srv736495810=84, srv2115889863=58, srv864457299=93, srv1013784908=0, srv932855811=98, srv2001918669=50, srv1071823776=4, srv1939139636=46, srv1159512086=9, srv1791647067=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:47,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:47,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:47,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:47,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:47,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:47,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-03 23:56:47,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-03 23:56:47,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-03 23:56:47,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-03 23:56:47,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-03 23:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-03 23:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-03 23:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-03 23:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-03 23:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-03 23:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-03 23:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-03 23:56:47,534 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,534 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-03 23:56:47,534 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.405228016640085E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-03 23:56:47,684 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 221 ms to try 5000 different iterations. Found a solution that moves 25 regions; Going from a computed imbalance of 3.405228016640085E-5 to a new imbalance of 7.864475541849422E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.002777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,685 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv436737872:83 , srv799065527:83 , srv913278986:83 , srv1013784908:84 , srv1036070409:84 , srv1044047461:84 , srv1064332542:84 , srv1071823776:84 , srv1083374500:84 , srv1102197047:84 , srv114438135:84 , srv1159512086:84 , srv1180773690:84 , srv1216945804:84 , srv125782310:84 , srv1275217638:84 , srv1275735693:84 , srv1287100506:84 , srv1289094034:84 , srv1293883056:84 , srv1319222276:84 , srv1345072397:84 , srv1371386428:84 , srv1411917757:84 , srv144158940:84 , srv1534283550:84 , srv1539590779:84 , srv1564967788:84 , srv1586135533:84 , srv1586664071:84 , srv1610358875:84 , srv1613204003:84 , srv1706710258:84 , srv1721103766:84 , srv1730914430:84 , srv174233906:84 , srv1743862113:84 , srv175829666:84 , srv1780001545:84 , srv1783820125:84 , srv1791647067:84 , srv1798604912:84 , srv1851324240:84 , srv1867450694:84 , srv1873148050:84 , srv1873500966:84 , srv1892046993:84 , srv192144871:84 , srv1939139636:84 , srv1942509372:84 , srv1975811962:84 , srv198186173:84 , srv2001918669:84 , srv2019001641:84 , srv2033576866:84 , srv2088377321:84 , srv2089218254:84 , srv2095361519:84 , srv2104040000:84 , srv2110911632:84 , srv2115889863:84 , srv2128011265:84 , srv220634207:84 , srv22499753:84 , srv232976865:84 , srv233436421:84 , srv243049872:84 , srv259035520:84 , srv268539222:84 , srv315288037:84 , srv325479144:84 , srv346829946:84 , srv423404518:84 , srv431921144:84 , srv43846102:84 , srv448202216:84 , srv466786590:84 , srv467385624:84 , srv483071537:84 , srv488053818:84 , srv558060772:84 , srv588828339:84 , srv603161703:84 , srv736495810:84 , srv764174477:84 , srv774719147:84 , srv798949527:84 , srv801028291:84 , srv833582759:84 , srv860866279:84 , srv864457299:84 , srv886601191:84 , srv90047612:84 , srv927728843:84 , srv932855811:84 , srv967514316:84 , srv1111917198:85 , srv529692366:85 , srv837043795:85 , srv671057643:684 } 2023-05-03 23:56:47,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:47,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv488053818=78, srv175829666=35, srv1534283550=23, srv268539222=66, srv1586135533=26, srv1975811962=48, srv588828339=81, srv114438135=8, srv1873500966=43, srv436737872=72, srv1275217638=13, srv174233906=33, srv220634207=60, srv798949527=87, srv1180773690=10, srv325479144=68, srv1867450694=41, srv1564967788=25, srv466786590=75, srv192144871=45, srv43846102=73, srv423404518=70, srv558060772=80, srv860866279=92, srv467385624=76, srv1293883056=17, srv90047612=95, srv1083374500=5, srv1345072397=19, srv144158940=22, srv1610358875=28, srv1586664071=27, srv1873148050=42, srv232976865=62, srv774719147=86, srv967514316=99, srv431921144=71, srv1613204003=29, srv1287100506=15, srv529692366=79, srv346829946=69, srv927728843=97, srv1743862113=34, srv1539590779=24, srv483071537=77, srv913278986=96, srv2019001641=51, srv1706710258=30, srv801028291=89, srv1319222276=18, srv1942509372=47, srv2128011265=59, srv603161703=82, srv1064332542=3, srv833582759=90, srv315288037=67, srv1411917757=21, srv1289094034=16, srv1102197047=6, srv1216945804=11, srv22499753=61, srv198186173=49, srv2110911632=57, srv1721103766=31, srv1275735693=14, srv1044047461=2, srv2033576866=52, srv2088377321=53, srv1371386428=20, srv1851324240=40, srv259035520=65, srv837043795=91, srv2095361519=55, srv1036070409=1, srv799065527=88, srv1892046993=44, srv886601191=94, srv1783820125=37, srv2089218254=54, srv448202216=74, srv1111917198=7, srv233436421=63, srv1780001545=36, srv243049872=64, srv671057643=83, srv1730914430=32, srv2104040000=56, srv764174477=85, srv125782310=12, srv1798604912=39, srv736495810=84, srv2115889863=58, srv864457299=93, srv1013784908=0, srv932855811=98, srv2001918669=50, srv1071823776=4, srv1939139636=46, srv1159512086=9, srv1791647067=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:47,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-03 23:56:47,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-03 23:56:47,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-03 23:56:47,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-03 23:56:47,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-03 23:56:47,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-03 23:56:47,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-03 23:56:47,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-03 23:56:47,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-03 23:56:47,777 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,777 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv436737872:83 , srv799065527:83 , srv913278986:83 , srv1013784908:84 , srv1036070409:84 , srv1044047461:84 , srv1064332542:84 , srv1071823776:84 , srv1083374500:84 , srv1102197047:84 , srv114438135:84 , srv1159512086:84 , srv1180773690:84 , srv1216945804:84 , srv125782310:84 , srv1275217638:84 , srv1275735693:84 , srv1287100506:84 , srv1289094034:84 , srv1293883056:84 , srv1319222276:84 , srv1345072397:84 , srv1371386428:84 , srv1411917757:84 , srv144158940:84 , srv1534283550:84 , srv1539590779:84 , srv1564967788:84 , srv1586135533:84 , srv1586664071:84 , srv1610358875:84 , srv1613204003:84 , srv1706710258:84 , srv1721103766:84 , srv1730914430:84 , srv174233906:84 , srv1743862113:84 , srv175829666:84 , srv1780001545:84 , srv1783820125:84 , srv1791647067:84 , srv1798604912:84 , srv1851324240:84 , srv1867450694:84 , srv1873148050:84 , srv1873500966:84 , srv1892046993:84 , srv192144871:84 , srv1939139636:84 , srv1942509372:84 , srv1975811962:84 , srv198186173:84 , srv2001918669:84 , srv2019001641:84 , srv2033576866:84 , srv2088377321:84 , srv2089218254:84 , srv2095361519:84 , srv2104040000:84 , srv2110911632:84 , srv2115889863:84 , srv2128011265:84 , srv220634207:84 , srv22499753:84 , srv232976865:84 , srv233436421:84 , srv243049872:84 , srv259035520:84 , srv268539222:84 , srv315288037:84 , srv325479144:84 , srv346829946:84 , srv423404518:84 , srv431921144:84 , srv43846102:84 , srv448202216:84 , srv466786590:84 , srv467385624:84 , srv483071537:84 , srv488053818:84 , srv558060772:84 , srv588828339:84 , srv603161703:84 , srv736495810:84 , srv764174477:84 , srv774719147:84 , srv798949527:84 , srv801028291:84 , srv833582759:84 , srv860866279:84 , srv864457299:84 , srv886601191:84 , srv90047612:84 , srv927728843:84 , srv932855811:84 , srv967514316:84 , srv1111917198:85 , srv529692366:85 , srv837043795:85 , srv671057643:684 } 2023-05-03 23:56:47,801 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=75 (was 75), ProcessCount=167 (was 167), AvailableMemoryMB=7863 (was 8349) 2023-05-03 23:56:47,808 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=75, ProcessCount=167, AvailableMemoryMB=7862 2023-05-03 23:56:47,808 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-03 23:56:47,808 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-03 23:56:47,816 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1124702025:84 , srv1223363106:84 , srv1223728457:84 , srv1254448811:84 , srv1379559670:84 , srv1407446357:84 , srv1443228586:84 , srv1624869088:84 , srv1650451270:84 , srv1734124291:84 , srv1739866826:84 , srv1760668626:84 , srv1812524019:84 , srv1890602056:84 , srv2008970242:84 , srv2102446385:84 , srv2131601863:84 , srv217661241:84 , srv253342071:84 , srv333461765:84 , srv377014163:84 , srv384889251:84 , srv435965285:84 , srv637092771:84 , srv767502219:84 , srv791746864:84 , srv841528100:84 , srv91989192:84 , srv97890769:84 , srv531717761:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-03 23:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812524019=12, srv333461765=19, srv2102446385=15, srv1443228586=6, srv1223728457=2, srv1734124291=9, srv531717761=23, srv637092771=24, srv2131601863=16, srv1650451270=8, srv1890602056=13, srv377014163=20, srv2008970242=14, srv1223363106=1, srv1760668626=11, srv1739866826=10, srv791746864=26, srv841528100=27, srv253342071=18, srv384889251=21, srv1124702025=0, srv767502219=25, srv1624869088=7, srv1254448811=3, srv435965285=22, srv1407446357=5, srv91989192=28, srv1379559670=4, srv217661241=17, srv97890769=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:56:47,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:56:47,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-03 23:56:47,840 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0863489365113934, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:56:47,840 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912465125943676, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0863489365113934, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-03 23:58:47,830 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9704068 different iterations. Found a solution that moves 1830 regions; Going from a computed imbalance of 0.9912465125943676 to a new imbalance of 4.271311292566818E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:58:47,838 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1124702025:90 , srv1223363106:90 , srv1223728457:90 , srv1254448811:90 , srv1379559670:90 , srv1407446357:90 , srv1443228586:90 , srv1624869088:90 , srv1650451270:90 , srv1734124291:90 , srv1739866826:90 , srv1760668626:90 , srv1812524019:90 , srv1890602056:90 , srv2008970242:90 , srv2102446385:90 , srv2131601863:90 , srv217661241:90 , srv253342071:90 , srv333461765:90 , srv377014163:90 , srv384889251:90 , srv435965285:90 , srv531717761:90 , srv637092771:90 , srv767502219:90 , srv791746864:90 , srv841528100:90 , srv91989192:90 , srv97890769:90 } 2023-05-03 23:58:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-03 23:58:47,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812524019=12, srv333461765=19, srv2102446385=15, srv1443228586=6, srv1223728457=2, srv1734124291=9, srv531717761=23, srv637092771=24, srv2131601863=16, srv1650451270=8, srv1890602056=13, srv377014163=20, srv2008970242=14, srv1223363106=1, srv1760668626=11, srv1739866826=10, srv791746864=26, srv841528100=27, srv253342071=18, srv384889251=21, srv1124702025=0, srv767502219=25, srv1624869088=7, srv1254448811=3, srv435965285=22, srv1407446357=5, srv91989192=28, srv1379559670=4, srv217661241=17, srv97890769=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-03 23:58:47,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-03 23:58:47,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-03 23:58:47,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-03 23:58:47,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-03 23:58:47,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-03 23:58:47,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-03 23:58:47,859 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-03 23:58:47,859 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1124702025:90 , srv1223363106:90 , srv1223728457:90 , srv1254448811:90 , srv1379559670:90 , srv1407446357:90 , srv1443228586:90 , srv1624869088:90 , srv1650451270:90 , srv1734124291:90 , srv1739866826:90 , srv1760668626:90 , srv1812524019:90 , srv1890602056:90 , srv2008970242:90 , srv2102446385:90 , srv2131601863:90 , srv217661241:90 , srv253342071:90 , srv333461765:90 , srv377014163:90 , srv384889251:90 , srv435965285:90 , srv531717761:90 , srv637092771:90 , srv767502219:90 , srv791746864:90 , srv841528100:90 , srv91989192:90 , srv97890769:90 } 2023-05-03 23:58:47,873 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=115 (was 75) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=8024 (was 7862) - AvailableMemoryMB LEAK? -