2024-04-21 07:53:34,839 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-21 07:53:35,136 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-21 07:53:35,241 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-21 07:53:35,298 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-21 07:53:35,306 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-21 07:53:35,320 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=263, ProcessCount=170, AvailableMemoryMB=7632 2024-04-21 07:53:35,322 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-21 07:53:35,322 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-21 07:53:35,436 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1035498050:84 , srv1044063893:84 , srv1055477243:84 , srv1083262052:84 , srv1093326224:84 , srv1108410017:84 , srv1145026764:84 , srv117440891:84 , srv1216757404:84 , srv1275032240:84 , srv1277813984:84 , srv1295678587:84 , srv1312339193:84 , srv1328197985:84 , srv1381567928:84 , srv1382371210:84 , srv1383665679:84 , srv1389583368:84 , srv1391374389:84 , srv1415776470:84 , srv1467453440:84 , srv1494533346:84 , srv1515903449:84 , srv1540220986:84 , srv1553535968:84 , srv1614185541:84 , srv1635614457:84 , srv1652528616:84 , srv166460065:84 , srv1682149110:84 , srv1702914337:84 , srv1712086498:84 , srv1835070238:84 , srv1845905471:84 , srv1848656117:84 , srv1850716422:84 , srv1851593458:84 , srv1882776295:84 , srv190954459:84 , srv1910668034:84 , srv1949515934:84 , srv1960292090:84 , srv1969118456:84 , srv1980481559:84 , srv2001832971:84 , srv2028279982:84 , srv2034979038:84 , srv2035742304:84 , srv2036516796:84 , srv2040986605:84 , srv2066361534:84 , srv2069122735:84 , srv212248976:84 , srv2134362908:84 , srv2139711934:84 , srv242110079:84 , srv266938561:84 , srv273721634:84 , srv277633905:84 , srv307463600:84 , srv341032250:84 , srv348005732:84 , srv36507249:84 , srv381220851:84 , srv383160634:84 , srv390266958:84 , srv39529427:84 , srv400985195:84 , srv401977674:84 , srv411740387:84 , srv412203642:84 , srv455701633:84 , srv491233093:84 , srv524090763:84 , srv538124763:84 , srv54342740:84 , srv547294349:84 , srv556173969:84 , srv572999687:84 , srv608084400:84 , srv665096023:84 , srv674321409:84 , srv683963710:84 , srv685685394:84 , srv704802753:84 , srv708613877:84 , srv711769687:84 , srv739973567:84 , srv757731780:84 , srv772893241:84 , srv791518286:84 , srv791917217:84 , srv804184140:84 , srv808292618:84 , srv809920640:84 , srv858931261:84 , srv913796652:84 , srv973419658:84 , srv997937256:84 , srv175603395:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-21 07:53:35,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:35,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2134362908=54, srv547294349=77, srv809920640=95, srv1553535968=24, srv1882776295=38, srv772893241=90, srv390266958=66, srv997937256=99, srv1035498050=0, srv190954459=39, srv1682149110=29, srv455701633=72, srv1494533346=21, srv1108410017=5, srv2139711934=55, srv175603395=32, srv2066361534=51, srv1949515934=41, srv572999687=79, srv608084400=80, srv711769687=87, srv1652528616=27, srv2028279982=46, srv36507249=63, srv1969118456=43, srv1382371210=15, srv400985195=68, srv1275032240=9, srv273721634=58, srv1312339193=12, srv412203642=71, srv2035742304=48, srv1835070238=33, srv556173969=78, srv1415776470=19, srv2034979038=47, srv166460065=28, srv212248976=53, srv1216757404=8, srv1277813984=10, srv401977674=69, srv1712086498=31, srv739973567=88, srv1540220986=23, srv266938561=57, srv674321409=82, srv1083262052=3, srv1702914337=30, srv685685394=84, srv1910668034=40, srv1383665679=16, srv411740387=70, srv341032250=61, srv1614185541=25, srv383160634=65, srv1295678587=11, srv307463600=60, srv683963710=83, srv665096023=81, srv1980481559=44, srv381220851=64, srv1851593458=37, srv858931261=96, srv1381567928=14, srv757731780=89, srv117440891=7, srv804184140=93, srv2040986605=50, srv538124763=75, srv1845905471=34, srv348005732=62, srv808292618=94, srv704802753=85, srv2069122735=52, srv1850716422=36, srv913796652=97, srv1635614457=26, srv708613877=86, srv1960292090=42, srv1044063893=1, srv1515903449=22, srv2001832971=45, srv54342740=76, srv1328197985=13, srv277633905=59, srv524090763=74, srv791917217=92, srv791518286=91, srv1093326224=4, srv1145026764=6, srv1389583368=17, srv2036516796=49, srv242110079=56, srv1848656117=35, srv1467453440=20, srv1391374389=18, srv973419658=98, srv1055477243=2, srv39529427=67, srv491233093=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:35,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:35,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:35,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:35,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:35,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-21 07:53:35,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-21 07:53:35,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-21 07:53:35,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-21 07:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-21 07:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-21 07:53:35,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-21 07:53:35,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-21 07:53:35,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-21 07:53:35,727 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:35,727 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-21 07:53:35,727 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-21 07:53:36,100 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 542 ms to try 5000 different iterations. Found a solution that moves 4672 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1791191062468713. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04050128589159457, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05308866437454376, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5191111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04050128589159457, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24358333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:36,117 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2134362908:80 , srv1093326224:81 , srv1553535968:82 , srv1969118456:82 , srv2028279982:82 , srv390266958:82 , srv1108410017:83 , srv1652528616:83 , srv491233093:83 , srv608084400:83 , srv1415776470:84 , srv1540220986:84 , srv1702914337:84 , srv1850716422:84 , srv266938561:84 , srv277633905:84 , srv400985195:84 , srv708613877:84 , srv757731780:84 , srv913796652:84 , srv1055477243:85 , srv1216757404:85 , srv1277813984:85 , srv1851593458:85 , srv190954459:85 , srv2036516796:85 , srv348005732:85 , srv383160634:85 , srv411740387:85 , srv412203642:85 , srv538124763:85 , srv665096023:85 , srv739973567:85 , srv808292618:85 , srv809920640:85 , srv997937256:85 , srv1381567928:86 , srv1515903449:86 , srv1682149110:86 , srv1949515934:86 , srv2001832971:86 , srv2034979038:86 , srv2035742304:86 , srv2040986605:86 , srv2066361534:86 , srv2139711934:86 , srv341032250:86 , srv455701633:86 , srv524090763:86 , srv54342740:86 , srv683963710:86 , srv791518286:86 , srv804184140:86 , srv973419658:86 , srv1083262052:87 , srv1295678587:87 , srv1328197985:87 , srv1383665679:87 , srv1467453440:87 , srv1614185541:87 , srv1635614457:87 , srv1848656117:87 , srv1910668034:87 , srv36507249:87 , srv381220851:87 , srv556173969:87 , srv572999687:87 , srv674321409:87 , srv711769687:87 , srv772893241:87 , srv1044063893:88 , srv117440891:88 , srv1391374389:88 , srv1845905471:88 , srv1882776295:88 , srv1980481559:88 , srv212248976:88 , srv242110079:88 , srv307463600:88 , srv401977674:88 , srv704802753:88 , srv1145026764:89 , srv1275032240:89 , srv1382371210:89 , srv1389583368:89 , srv1494533346:89 , srv1712086498:89 , srv1835070238:89 , srv1960292090:89 , srv2069122735:89 , srv273721634:89 , srv39529427:89 , srv791917217:89 , srv1035498050:90 , srv547294349:91 , srv858931261:91 , srv1312339193:92 , srv166460065:92 , srv685685394:97 , srv175603395:450 } 2024-04-21 07:53:36,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:36,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2134362908=54, srv547294349=77, srv809920640=95, srv1553535968=24, srv1882776295=38, srv772893241=90, srv390266958=66, srv997937256=99, srv1035498050=0, srv190954459=39, srv1682149110=29, srv455701633=72, srv1494533346=21, srv1108410017=5, srv2139711934=55, srv175603395=32, srv2066361534=51, srv1949515934=41, srv572999687=79, srv608084400=80, srv711769687=87, srv1652528616=27, srv2028279982=46, srv36507249=63, srv1969118456=43, srv1382371210=15, srv400985195=68, srv1275032240=9, srv273721634=58, srv1312339193=12, srv412203642=71, srv2035742304=48, srv1835070238=33, srv556173969=78, srv1415776470=19, srv2034979038=47, srv166460065=28, srv212248976=53, srv1216757404=8, srv1277813984=10, srv401977674=69, srv1712086498=31, srv739973567=88, srv1540220986=23, srv266938561=57, srv674321409=82, srv1083262052=3, srv1702914337=30, srv685685394=84, srv1910668034=40, srv1383665679=16, srv411740387=70, srv341032250=61, srv1614185541=25, srv383160634=65, srv1295678587=11, srv307463600=60, srv683963710=83, srv665096023=81, srv1980481559=44, srv381220851=64, srv1851593458=37, srv858931261=96, srv1381567928=14, srv757731780=89, srv117440891=7, srv804184140=93, srv2040986605=50, srv538124763=75, srv1845905471=34, srv348005732=62, srv808292618=94, srv704802753=85, srv2069122735=52, srv1850716422=36, srv913796652=97, srv1635614457=26, srv708613877=86, srv1960292090=42, srv1044063893=1, srv1515903449=22, srv2001832971=45, srv54342740=76, srv1328197985=13, srv277633905=59, srv524090763=74, srv791917217=92, srv791518286=91, srv1093326224=4, srv1145026764=6, srv1389583368=17, srv2036516796=49, srv242110079=56, srv1848656117=35, srv1467453440=20, srv1391374389=18, srv973419658=98, srv1055477243=2, srv39529427=67, srv491233093=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:36,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:36,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-21 07:53:36,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-21 07:53:36,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-21 07:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-21 07:53:36,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-21 07:53:36,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-21 07:53:36,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-21 07:53:36,265 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04050128589159457, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05308866437454376, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04050128589159457, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24358333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:36,265 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-21 07:53:36,265 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17908639220366, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04050128589159457, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05308866437454376, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04050128589159457, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24358333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-21 07:53:36,567 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations. Found a solution that moves 2603 regions; Going from a computed imbalance of 0.17908639220366 to a new imbalance of 0.016448140981545827. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022404972458623548); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03599787997249323, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2892222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022404972458623548); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048666666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:36,580 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv685685394:74 , srv547294349:79 , srv858931261:80 , srv166460065:81 , srv1712086498:81 , srv1835070238:81 , srv791917217:81 , srv1035498050:82 , srv1275032240:82 , srv307463600:82 , srv39529427:82 , srv572999687:82 , srv1044063893:83 , srv1145026764:83 , srv1312339193:83 , srv1382371210:83 , srv1383665679:83 , srv1494533346:83 , srv1949515934:83 , srv1960292090:83 , srv1980481559:83 , srv2001832971:83 , srv2066361534:83 , srv2069122735:83 , srv273721634:83 , srv772893241:83 , srv1328197985:84 , srv1389583368:84 , srv1614185541:84 , srv1882776295:84 , srv804184140:84 , srv117440891:85 , srv1381567928:85 , srv1391374389:85 , srv1682149110:85 , srv1848656117:85 , srv1851593458:85 , srv1910668034:85 , srv2035742304:85 , srv2036516796:85 , srv36507249:85 , srv381220851:85 , srv455701633:85 , srv556173969:85 , srv674321409:85 , srv791518286:85 , srv973419658:85 , srv1083262052:86 , srv1277813984:86 , srv1467453440:86 , srv1635614457:86 , srv1702914337:86 , srv2034979038:86 , srv2040986605:86 , srv2139711934:86 , srv242110079:86 , srv266938561:86 , srv54342740:86 , srv711769687:86 , srv739973567:86 , srv809920640:86 , srv1055477243:87 , srv1515903449:87 , srv1845905471:87 , srv383160634:87 , srv400985195:87 , srv401977674:87 , srv524090763:87 , srv683963710:87 , srv808292618:87 , srv913796652:87 , srv1216757404:88 , srv1295678587:88 , srv1415776470:88 , srv1850716422:88 , srv1969118456:88 , srv212248976:88 , srv348005732:88 , srv411740387:88 , srv491233093:88 , srv538124763:88 , srv665096023:88 , srv704802753:88 , srv757731780:88 , srv1108410017:89 , srv190954459:89 , srv2028279982:89 , srv277633905:89 , srv341032250:89 , srv390266958:89 , srv412203642:89 , srv708613877:89 , srv997937256:90 , srv1093326224:91 , srv1540220986:91 , srv2134362908:91 , srv608084400:91 , srv1553535968:92 , srv1652528616:92 , srv175603395:523 } 2024-04-21 07:53:36,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:36,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2134362908=54, srv547294349=77, srv809920640=95, srv1553535968=24, srv1882776295=38, srv772893241=90, srv390266958=66, srv997937256=99, srv1035498050=0, srv190954459=39, srv1682149110=29, srv455701633=72, srv1494533346=21, srv1108410017=5, srv2139711934=55, srv175603395=32, srv2066361534=51, srv1949515934=41, srv572999687=79, srv608084400=80, srv711769687=87, srv1652528616=27, srv2028279982=46, srv36507249=63, srv1969118456=43, srv1382371210=15, srv400985195=68, srv1275032240=9, srv273721634=58, srv1312339193=12, srv412203642=71, srv2035742304=48, srv1835070238=33, srv556173969=78, srv1415776470=19, srv2034979038=47, srv166460065=28, srv212248976=53, srv1216757404=8, srv1277813984=10, srv401977674=69, srv1712086498=31, srv739973567=88, srv1540220986=23, srv266938561=57, srv674321409=82, srv1083262052=3, srv1702914337=30, srv685685394=84, srv1910668034=40, srv1383665679=16, srv411740387=70, srv341032250=61, srv1614185541=25, srv383160634=65, srv1295678587=11, srv307463600=60, srv683963710=83, srv665096023=81, srv1980481559=44, srv381220851=64, srv1851593458=37, srv858931261=96, srv1381567928=14, srv757731780=89, srv117440891=7, srv804184140=93, srv2040986605=50, srv538124763=75, srv1845905471=34, srv348005732=62, srv808292618=94, srv704802753=85, srv2069122735=52, srv1850716422=36, srv913796652=97, srv1635614457=26, srv708613877=86, srv1960292090=42, srv1044063893=1, srv1515903449=22, srv2001832971=45, srv54342740=76, srv1328197985=13, srv277633905=59, srv524090763=74, srv791917217=92, srv791518286=91, srv1093326224=4, srv1145026764=6, srv1389583368=17, srv2036516796=49, srv242110079=56, srv1848656117=35, srv1467453440=20, srv1391374389=18, srv973419658=98, srv1055477243=2, srv39529427=67, srv491233093=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:36,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:36,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:36,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-21 07:53:36,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-21 07:53:36,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-21 07:53:36,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-21 07:53:36,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-21 07:53:36,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-21 07:53:36,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-21 07:53:36,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-21 07:53:36,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-21 07:53:36,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-21 07:53:36,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-21 07:53:36,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-21 07:53:36,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-21 07:53:36,753 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022404972458623548); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03599787997249323, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022404972458623548); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048666666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:36,753 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-21 07:53:36,754 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01642991438598099, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022404972458623548); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03599787997249323, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022404972458623548); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048666666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-21 07:53:37,169 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 538 ms to try 5000 different iterations. Found a solution that moves 982 regions; Going from a computed imbalance of 0.01642991438598099 to a new imbalance of 0.0011899459413958918. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01282708974011244); MoveCostFunction : (multiplier=7.0, imbalance=0.10911111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007784989441615229); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:37,171 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv212248976:81 , srv704802753:81 , srv997937256:81 , srv1540220986:82 , srv1845905471:82 , srv341032250:82 , srv401977674:82 , srv1312339193:83 , srv1553535968:83 , srv190954459:83 , srv242110079:83 , srv412203642:83 , srv1083262052:84 , srv1216757404:84 , srv1295678587:84 , srv1389583368:84 , srv1391374389:84 , srv1467453440:84 , srv1515903449:84 , srv1635614457:84 , srv1652528616:84 , srv277633905:84 , srv348005732:84 , srv524090763:84 , srv608084400:84 , srv665096023:84 , srv708613877:84 , srv711769687:84 , srv1035498050:85 , srv1055477243:85 , srv1108410017:85 , srv1145026764:85 , srv117440891:85 , srv1275032240:85 , srv1382371210:85 , srv166460065:85 , srv1682149110:85 , srv1848656117:85 , srv1850716422:85 , srv1882776295:85 , srv1960292090:85 , srv2034979038:85 , srv2040986605:85 , srv2069122735:85 , srv2139711934:85 , srv381220851:85 , srv383160634:85 , srv411740387:85 , srv674321409:85 , srv683963710:85 , srv973419658:85 , srv1044063893:86 , srv1277813984:86 , srv1328197985:86 , srv1381567928:86 , srv1415776470:86 , srv1494533346:86 , srv1614185541:86 , srv1712086498:86 , srv1910668034:86 , srv1980481559:86 , srv2028279982:86 , srv2035742304:86 , srv2134362908:86 , srv273721634:86 , srv36507249:86 , srv400985195:86 , srv538124763:86 , srv54342740:86 , srv757731780:86 , srv791518286:86 , srv804184140:86 , srv809920640:86 , srv858931261:86 , srv913796652:86 , srv1093326224:87 , srv1383665679:87 , srv1851593458:87 , srv1969118456:87 , srv2066361534:87 , srv266938561:87 , srv307463600:87 , srv390266958:87 , srv455701633:87 , srv491233093:87 , srv547294349:87 , srv556173969:87 , srv685685394:87 , srv772893241:87 , srv791917217:87 , srv808292618:87 , srv1702914337:88 , srv1835070238:88 , srv2001832971:88 , srv2036516796:88 , srv39529427:88 , srv572999687:88 , srv739973567:88 , srv1949515934:89 , srv175603395:554 } 2024-04-21 07:53:37,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:37,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2134362908=54, srv547294349=77, srv809920640=95, srv1553535968=24, srv1882776295=38, srv772893241=90, srv390266958=66, srv997937256=99, srv1035498050=0, srv190954459=39, srv1682149110=29, srv455701633=72, srv1494533346=21, srv1108410017=5, srv2139711934=55, srv175603395=32, srv2066361534=51, srv1949515934=41, srv572999687=79, srv608084400=80, srv711769687=87, srv1652528616=27, srv2028279982=46, srv36507249=63, srv1969118456=43, srv1382371210=15, srv400985195=68, srv1275032240=9, srv273721634=58, srv1312339193=12, srv412203642=71, srv2035742304=48, srv1835070238=33, srv556173969=78, srv1415776470=19, srv2034979038=47, srv166460065=28, srv212248976=53, srv1216757404=8, srv1277813984=10, srv401977674=69, srv1712086498=31, srv739973567=88, srv1540220986=23, srv266938561=57, srv674321409=82, srv1083262052=3, srv1702914337=30, srv685685394=84, srv1910668034=40, srv1383665679=16, srv411740387=70, srv341032250=61, srv1614185541=25, srv383160634=65, srv1295678587=11, srv307463600=60, srv683963710=83, srv665096023=81, srv1980481559=44, srv381220851=64, srv1851593458=37, srv858931261=96, srv1381567928=14, srv757731780=89, srv117440891=7, srv804184140=93, srv2040986605=50, srv538124763=75, srv1845905471=34, srv348005732=62, srv808292618=94, srv704802753=85, srv2069122735=52, srv1850716422=36, srv913796652=97, srv1635614457=26, srv708613877=86, srv1960292090=42, srv1044063893=1, srv1515903449=22, srv2001832971=45, srv54342740=76, srv1328197985=13, srv277633905=59, srv524090763=74, srv791917217=92, srv791518286=91, srv1093326224=4, srv1145026764=6, srv1389583368=17, srv2036516796=49, srv242110079=56, srv1848656117=35, srv1467453440=20, srv1391374389=18, srv973419658=98, srv1055477243=2, srv39529427=67, srv491233093=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:37,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:37,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:37,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:37,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:37,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-21 07:53:37,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-21 07:53:37,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-21 07:53:37,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-21 07:53:37,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-21 07:53:37,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-21 07:53:37,301 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01282708974011244); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007784989441615229); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:37,301 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-21 07:53:37,301 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.001183069830429825, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01282708974011244); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007784989441615229); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-21 07:53:37,570 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 367 ms to try 5000 different iterations. Found a solution that moves 283 regions; Going from a computed imbalance of 0.001183069830429825 to a new imbalance of 3.5490842403134846E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:37,573 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1093326224:82 , srv1652528616:83 , srv1910668034:83 , srv39529427:83 , srv455701633:83 , srv556173969:83 , srv608084400:83 , srv739973567:83 , srv1035498050:84 , srv117440891:84 , srv1216757404:84 , srv1295678587:84 , srv1312339193:84 , srv1383665679:84 , srv1415776470:84 , srv1494533346:84 , srv1553535968:84 , srv166460065:84 , srv1835070238:84 , srv1949515934:84 , srv1980481559:84 , srv2035742304:84 , srv2040986605:84 , srv2069122735:84 , srv273721634:84 , srv36507249:84 , srv538124763:84 , srv54342740:84 , srv683963710:84 , srv685685394:84 , srv757731780:84 , srv808292618:84 , srv1044063893:85 , srv1055477243:85 , srv1083262052:85 , srv1108410017:85 , srv1145026764:85 , srv1275032240:85 , srv1277813984:85 , srv1328197985:85 , srv1381567928:85 , srv1382371210:85 , srv1389583368:85 , srv1391374389:85 , srv1467453440:85 , srv1515903449:85 , srv1540220986:85 , srv1614185541:85 , srv1635614457:85 , srv1702914337:85 , srv1845905471:85 , srv1848656117:85 , srv1850716422:85 , srv1851593458:85 , srv1882776295:85 , srv190954459:85 , srv1969118456:85 , srv2001832971:85 , srv2034979038:85 , srv2036516796:85 , srv212248976:85 , srv2134362908:85 , srv2139711934:85 , srv242110079:85 , srv266938561:85 , srv277633905:85 , srv307463600:85 , srv341032250:85 , srv381220851:85 , srv383160634:85 , srv390266958:85 , srv400985195:85 , srv401977674:85 , srv411740387:85 , srv412203642:85 , srv491233093:85 , srv524090763:85 , srv547294349:85 , srv665096023:85 , srv674321409:85 , srv704802753:85 , srv708613877:85 , srv711769687:85 , srv772893241:85 , srv791518286:85 , srv791917217:85 , srv809920640:85 , srv858931261:85 , srv913796652:85 , srv1682149110:86 , srv1960292090:86 , srv2028279982:86 , srv2066361534:86 , srv348005732:86 , srv572999687:86 , srv804184140:86 , srv973419658:86 , srv997937256:86 , srv1712086498:87 , srv175603395:615 } 2024-04-21 07:53:37,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:37,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2134362908=54, srv547294349=77, srv809920640=95, srv1553535968=24, srv1882776295=38, srv772893241=90, srv390266958=66, srv997937256=99, srv1035498050=0, srv190954459=39, srv1682149110=29, srv455701633=72, srv1494533346=21, srv1108410017=5, srv2139711934=55, srv175603395=32, srv2066361534=51, srv1949515934=41, srv572999687=79, srv608084400=80, srv711769687=87, srv1652528616=27, srv2028279982=46, srv36507249=63, srv1969118456=43, srv1382371210=15, srv400985195=68, srv1275032240=9, srv273721634=58, srv1312339193=12, srv412203642=71, srv2035742304=48, srv1835070238=33, srv556173969=78, srv1415776470=19, srv2034979038=47, srv166460065=28, srv212248976=53, srv1216757404=8, srv1277813984=10, srv401977674=69, srv1712086498=31, srv739973567=88, srv1540220986=23, srv266938561=57, srv674321409=82, srv1083262052=3, srv1702914337=30, srv685685394=84, srv1910668034=40, srv1383665679=16, srv411740387=70, srv341032250=61, srv1614185541=25, srv383160634=65, srv1295678587=11, srv307463600=60, srv683963710=83, srv665096023=81, srv1980481559=44, srv381220851=64, srv1851593458=37, srv858931261=96, srv1381567928=14, srv757731780=89, srv117440891=7, srv804184140=93, srv2040986605=50, srv538124763=75, srv1845905471=34, srv348005732=62, srv808292618=94, srv704802753=85, srv2069122735=52, srv1850716422=36, srv913796652=97, srv1635614457=26, srv708613877=86, srv1960292090=42, srv1044063893=1, srv1515903449=22, srv2001832971=45, srv54342740=76, srv1328197985=13, srv277633905=59, srv524090763=74, srv791917217=92, srv791518286=91, srv1093326224=4, srv1145026764=6, srv1389583368=17, srv2036516796=49, srv242110079=56, srv1848656117=35, srv1467453440=20, srv1391374389=18, srv973419658=98, srv1055477243=2, srv39529427=67, srv491233093=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:37,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:37,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:37,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:37,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:37,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:37,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:37,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-21 07:53:37,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-21 07:53:37,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-21 07:53:37,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-21 07:53:37,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-21 07:53:37,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-21 07:53:37,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-21 07:53:37,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-21 07:53:37,746 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:37,746 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-21 07:53:37,747 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.3509234049370244E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-21 07:53:38,031 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 422 ms to try 5000 different iterations. Found a solution that moves 37 regions; Going from a computed imbalance of 3.3509234049370244E-5 to a new imbalance of 5.901572530944662E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:38,031 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1295678587:83 , srv1702914337:83 , srv1712086498:83 , srv1960292090:83 , srv2028279982:83 , srv2036516796:83 , srv348005732:83 , srv390266958:83 , srv411740387:83 , srv491233093:83 , srv538124763:83 , srv572999687:83 , srv808292618:83 , srv1044063893:84 , srv1055477243:84 , srv1083262052:84 , srv1108410017:84 , srv1145026764:84 , srv117440891:84 , srv1277813984:84 , srv1312339193:84 , srv1328197985:84 , srv1381567928:84 , srv1382371210:84 , srv1389583368:84 , srv1391374389:84 , srv1415776470:84 , srv1467453440:84 , srv1494533346:84 , srv1515903449:84 , srv1540220986:84 , srv1614185541:84 , srv1635614457:84 , srv1652528616:84 , srv166460065:84 , srv1682149110:84 , srv1835070238:84 , srv1845905471:84 , srv1848656117:84 , srv1850716422:84 , srv1851593458:84 , srv1882776295:84 , srv190954459:84 , srv1949515934:84 , srv1969118456:84 , srv2001832971:84 , srv2034979038:84 , srv2066361534:84 , srv212248976:84 , srv2134362908:84 , srv2139711934:84 , srv242110079:84 , srv266938561:84 , srv273721634:84 , srv277633905:84 , srv307463600:84 , srv341032250:84 , srv36507249:84 , srv381220851:84 , srv383160634:84 , srv39529427:84 , srv400985195:84 , srv401977674:84 , srv412203642:84 , srv524090763:84 , srv54342740:84 , srv547294349:84 , srv556173969:84 , srv665096023:84 , srv674321409:84 , srv683963710:84 , srv685685394:84 , srv704802753:84 , srv708613877:84 , srv711769687:84 , srv739973567:84 , srv757731780:84 , srv772893241:84 , srv791518286:84 , srv791917217:84 , srv804184140:84 , srv809920640:84 , srv858931261:84 , srv913796652:84 , srv973419658:84 , srv997937256:84 , srv1035498050:85 , srv1093326224:85 , srv1216757404:85 , srv1275032240:85 , srv1383665679:85 , srv1553535968:85 , srv1910668034:85 , srv1980481559:85 , srv2035742304:85 , srv2040986605:85 , srv2069122735:85 , srv455701633:85 , srv608084400:85 , srv175603395:684 } 2024-04-21 07:53:38,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:38,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2134362908=54, srv547294349=77, srv809920640=95, srv1553535968=24, srv1882776295=38, srv772893241=90, srv390266958=66, srv997937256=99, srv1035498050=0, srv190954459=39, srv1682149110=29, srv455701633=72, srv1494533346=21, srv1108410017=5, srv2139711934=55, srv175603395=32, srv2066361534=51, srv1949515934=41, srv572999687=79, srv608084400=80, srv711769687=87, srv1652528616=27, srv2028279982=46, srv36507249=63, srv1969118456=43, srv1382371210=15, srv400985195=68, srv1275032240=9, srv273721634=58, srv1312339193=12, srv412203642=71, srv2035742304=48, srv1835070238=33, srv556173969=78, srv1415776470=19, srv2034979038=47, srv166460065=28, srv212248976=53, srv1216757404=8, srv1277813984=10, srv401977674=69, srv1712086498=31, srv739973567=88, srv1540220986=23, srv266938561=57, srv674321409=82, srv1083262052=3, srv1702914337=30, srv685685394=84, srv1910668034=40, srv1383665679=16, srv411740387=70, srv341032250=61, srv1614185541=25, srv383160634=65, srv1295678587=11, srv307463600=60, srv683963710=83, srv665096023=81, srv1980481559=44, srv381220851=64, srv1851593458=37, srv858931261=96, srv1381567928=14, srv757731780=89, srv117440891=7, srv804184140=93, srv2040986605=50, srv538124763=75, srv1845905471=34, srv348005732=62, srv808292618=94, srv704802753=85, srv2069122735=52, srv1850716422=36, srv913796652=97, srv1635614457=26, srv708613877=86, srv1960292090=42, srv1044063893=1, srv1515903449=22, srv2001832971=45, srv54342740=76, srv1328197985=13, srv277633905=59, srv524090763=74, srv791917217=92, srv791518286=91, srv1093326224=4, srv1145026764=6, srv1389583368=17, srv2036516796=49, srv242110079=56, srv1848656117=35, srv1467453440=20, srv1391374389=18, srv973419658=98, srv1055477243=2, srv39529427=67, srv491233093=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:38,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-21 07:53:38,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-21 07:53:38,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-21 07:53:38,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-21 07:53:38,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-21 07:53:38,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-21 07:53:38,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-21 07:53:38,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-21 07:53:38,162 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.642492993526674E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:38,163 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1295678587:83 , srv1702914337:83 , srv1712086498:83 , srv1960292090:83 , srv2028279982:83 , srv2036516796:83 , srv348005732:83 , srv390266958:83 , srv411740387:83 , srv491233093:83 , srv538124763:83 , srv572999687:83 , srv808292618:83 , srv1044063893:84 , srv1055477243:84 , srv1083262052:84 , srv1108410017:84 , srv1145026764:84 , srv117440891:84 , srv1277813984:84 , srv1312339193:84 , srv1328197985:84 , srv1381567928:84 , srv1382371210:84 , srv1389583368:84 , srv1391374389:84 , srv1415776470:84 , srv1467453440:84 , srv1494533346:84 , srv1515903449:84 , srv1540220986:84 , srv1614185541:84 , srv1635614457:84 , srv1652528616:84 , srv166460065:84 , srv1682149110:84 , srv1835070238:84 , srv1845905471:84 , srv1848656117:84 , srv1850716422:84 , srv1851593458:84 , srv1882776295:84 , srv190954459:84 , srv1949515934:84 , srv1969118456:84 , srv2001832971:84 , srv2034979038:84 , srv2066361534:84 , srv212248976:84 , srv2134362908:84 , srv2139711934:84 , srv242110079:84 , srv266938561:84 , srv273721634:84 , srv277633905:84 , srv307463600:84 , srv341032250:84 , srv36507249:84 , srv381220851:84 , srv383160634:84 , srv39529427:84 , srv400985195:84 , srv401977674:84 , srv412203642:84 , srv524090763:84 , srv54342740:84 , srv547294349:84 , srv556173969:84 , srv665096023:84 , srv674321409:84 , srv683963710:84 , srv685685394:84 , srv704802753:84 , srv708613877:84 , srv711769687:84 , srv739973567:84 , srv757731780:84 , srv772893241:84 , srv791518286:84 , srv791917217:84 , srv804184140:84 , srv809920640:84 , srv858931261:84 , srv913796652:84 , srv973419658:84 , srv997937256:84 , srv1035498050:85 , srv1093326224:85 , srv1216757404:85 , srv1275032240:85 , srv1383665679:85 , srv1553535968:85 , srv1910668034:85 , srv1980481559:85 , srv2035742304:85 , srv2040986605:85 , srv2069122735:85 , srv455701633:85 , srv608084400:85 , srv175603395:684 } 2024-04-21 07:53:38,191 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=263 (was 263), ProcessCount=170 (was 170), AvailableMemoryMB=7072 (was 7632) 2024-04-21 07:53:38,199 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=263, ProcessCount=170, AvailableMemoryMB=7072 2024-04-21 07:53:38,200 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-21 07:53:38,200 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-21 07:53:38,210 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1025442131:84 , srv1110447920:84 , srv1147278253:84 , srv1246333572:84 , srv126458796:84 , srv1310361995:84 , srv1319042595:84 , srv1375594550:84 , srv1398242034:84 , srv1434340479:84 , srv1452319384:84 , srv1458523190:84 , srv1466885404:84 , srv1529193034:84 , srv1594043346:84 , srv1651099708:84 , srv1688929958:84 , srv1695164187:84 , srv1757100685:84 , srv1799238771:84 , srv183146202:84 , srv2117423594:84 , srv335497541:84 , srv619695027:84 , srv622460266:84 , srv667318761:84 , srv742075257:84 , srv768875086:84 , srv963561201:84 , srv1040449624:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-21 07:53:38,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:53:38,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1398242034=9, srv667318761=26, srv1594043346=15, srv768875086=28, srv1695164187=18, srv1147278253=3, srv619695027=24, srv622460266=25, srv1040449624=1, srv1651099708=16, srv1799238771=20, srv1458523190=12, srv742075257=27, srv1110447920=2, srv1466885404=13, srv126458796=5, srv1688929958=17, srv1319042595=7, srv1452319384=11, srv1025442131=0, srv2117423594=22, srv1757100685=19, srv1310361995=6, srv1434340479=10, srv1529193034=14, srv963561201=29, srv1375594550=8, srv183146202=21, srv1246333572=4, srv335497541=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:53:38,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:53:38,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:53:38,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:53:38,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:53:38,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:53:38,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:53:38,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:53:38,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:53:38,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:53:38,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-21 07:53:38,241 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0880376569300987, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:53:38,241 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912470447046573, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0880376569300987, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-21 07:55:38,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6030977 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912470447046573 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:55:38,236 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1025442131:90 , srv1040449624:90 , srv1110447920:90 , srv1147278253:90 , srv1246333572:90 , srv126458796:90 , srv1310361995:90 , srv1319042595:90 , srv1375594550:90 , srv1398242034:90 , srv1434340479:90 , srv1452319384:90 , srv1458523190:90 , srv1466885404:90 , srv1529193034:90 , srv1594043346:90 , srv1651099708:90 , srv1688929958:90 , srv1695164187:90 , srv1757100685:90 , srv1799238771:90 , srv183146202:90 , srv2117423594:90 , srv335497541:90 , srv619695027:90 , srv622460266:90 , srv667318761:90 , srv742075257:90 , srv768875086:90 , srv963561201:90 } 2024-04-21 07:55:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-21 07:55:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1398242034=9, srv667318761=26, srv1594043346=15, srv768875086=28, srv1695164187=18, srv1147278253=3, srv619695027=24, srv622460266=25, srv1040449624=1, srv1651099708=16, srv1799238771=20, srv1458523190=12, srv742075257=27, srv1110447920=2, srv1466885404=13, srv126458796=5, srv1688929958=17, srv1319042595=7, srv1452319384=11, srv1025442131=0, srv2117423594=22, srv1757100685=19, srv1310361995=6, srv1434340479=10, srv1529193034=14, srv963561201=29, srv1375594550=8, srv183146202=21, srv1246333572=4, srv335497541=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-21 07:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-21 07:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-21 07:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-21 07:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-21 07:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-21 07:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-21 07:55:38,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-21 07:55:38,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-21 07:55:38,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-21 07:55:38,257 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-21 07:55:38,257 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1025442131:90 , srv1040449624:90 , srv1110447920:90 , srv1147278253:90 , srv1246333572:90 , srv126458796:90 , srv1310361995:90 , srv1319042595:90 , srv1375594550:90 , srv1398242034:90 , srv1434340479:90 , srv1452319384:90 , srv1458523190:90 , srv1466885404:90 , srv1529193034:90 , srv1594043346:90 , srv1651099708:90 , srv1688929958:90 , srv1695164187:90 , srv1757100685:90 , srv1799238771:90 , srv183146202:90 , srv2117423594:90 , srv335497541:90 , srv619695027:90 , srv622460266:90 , srv667318761:90 , srv742075257:90 , srv768875086:90 , srv963561201:90 } 2024-04-21 07:55:38,271 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=180 (was 263), ProcessCount=167 (was 170), AvailableMemoryMB=7045 (was 7072)