2023-07-15 18:12:51,811 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0 2023-07-15 18:12:51,831 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.rsgroup.TestRSGroupsFallback timeout: 13 mins 2023-07-15 18:12:51,860 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-15 18:12:51,860 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a, deleteOnExit=true 2023-07-15 18:12:51,861 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-15 18:12:51,861 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/test.cache.data in system properties and HBase conf 2023-07-15 18:12:51,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.tmp.dir in system properties and HBase conf 2023-07-15 18:12:51,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir in system properties and HBase conf 2023-07-15 18:12:51,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-15 18:12:51,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-15 18:12:51,864 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-15 18:12:52,056 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-15 18:12:52,593 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-15 18:12:52,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-15 18:12:52,600 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-15 18:12:52,600 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-15 18:12:52,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 18:12:52,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-15 18:12:52,601 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-15 18:12:52,602 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 18:12:52,602 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 18:12:52,603 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-15 18:12:52,603 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/nfs.dump.dir in system properties and HBase conf 2023-07-15 18:12:52,603 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/java.io.tmpdir in system properties and HBase conf 2023-07-15 18:12:52,604 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 18:12:52,604 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-15 18:12:52,604 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-15 18:12:53,293 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 18:12:53,298 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 18:12:53,682 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-15 18:12:53,961 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-15 18:12:53,987 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 18:12:54,039 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 18:12:54,088 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/java.io.tmpdir/Jetty_localhost_33629_hdfs____s5pxl3/webapp 2023-07-15 18:12:54,310 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33629 2023-07-15 18:12:54,363 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 18:12:54,364 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 18:12:55,038 WARN [Listener at localhost/39481] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 18:12:55,334 WARN [Listener at localhost/39481] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 18:12:55,369 WARN [Listener at localhost/39481] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 18:12:55,377 INFO [Listener at localhost/39481] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 18:12:55,386 INFO [Listener at localhost/39481] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/java.io.tmpdir/Jetty_localhost_39859_datanode____sqzijh/webapp 2023-07-15 18:12:55,526 INFO [Listener at localhost/39481] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39859 2023-07-15 18:12:56,276 WARN [Listener at localhost/46875] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 18:12:56,398 WARN [Listener at localhost/46875] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 18:12:56,402 WARN [Listener at localhost/46875] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 18:12:56,405 INFO [Listener at localhost/46875] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 18:12:56,432 INFO [Listener at localhost/46875] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/java.io.tmpdir/Jetty_localhost_45889_datanode____.b7jytd/webapp 2023-07-15 18:12:56,546 INFO [Listener at localhost/46875] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45889 2023-07-15 18:12:56,556 WARN [Listener at localhost/41107] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 18:12:56,591 WARN [Listener at localhost/41107] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 18:12:56,594 WARN [Listener at localhost/41107] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 18:12:56,596 INFO [Listener at localhost/41107] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 18:12:56,608 INFO [Listener at localhost/41107] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/java.io.tmpdir/Jetty_localhost_34129_datanode____.7g0ode/webapp 2023-07-15 18:12:56,768 INFO [Listener at localhost/41107] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34129 2023-07-15 18:12:56,825 WARN [Listener at localhost/46109] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 18:12:57,236 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x18dfa5fd0ccdfff1: Processing first storage report for DS-b864938b-2b4e-42e3-bc03-41329159ae0d from datanode 83b629db-5560-495e-b685-389dfc1aa428 2023-07-15 18:12:57,238 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x18dfa5fd0ccdfff1: from storage DS-b864938b-2b4e-42e3-bc03-41329159ae0d node DatanodeRegistration(127.0.0.1:37907, datanodeUuid=83b629db-5560-495e-b685-389dfc1aa428, infoPort=36073, infoSecurePort=0, ipcPort=46875, storageInfo=lv=-57;cid=testClusterID;nsid=2007964022;c=1689444773408), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-15 18:12:57,238 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x28b7867468f8bb90: Processing first storage report for DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd from datanode 8050b55f-8f12-47d7-a080-9ddfbceb52d3 2023-07-15 18:12:57,238 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x28b7867468f8bb90: from storage DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd node DatanodeRegistration(127.0.0.1:45437, datanodeUuid=8050b55f-8f12-47d7-a080-9ddfbceb52d3, infoPort=46017, infoSecurePort=0, ipcPort=41107, storageInfo=lv=-57;cid=testClusterID;nsid=2007964022;c=1689444773408), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 18:12:57,238 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x18dfa5fd0ccdfff1: Processing first storage report for DS-1168d8fb-9516-41b1-9a7b-7467e75557f9 from datanode 83b629db-5560-495e-b685-389dfc1aa428 2023-07-15 18:12:57,238 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x18dfa5fd0ccdfff1: from storage DS-1168d8fb-9516-41b1-9a7b-7467e75557f9 node DatanodeRegistration(127.0.0.1:37907, datanodeUuid=83b629db-5560-495e-b685-389dfc1aa428, infoPort=36073, infoSecurePort=0, ipcPort=46875, storageInfo=lv=-57;cid=testClusterID;nsid=2007964022;c=1689444773408), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 18:12:57,239 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x28b7867468f8bb90: Processing first storage report for DS-cd5de5bc-aae6-4c26-afd7-58c896927415 from datanode 8050b55f-8f12-47d7-a080-9ddfbceb52d3 2023-07-15 18:12:57,239 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x28b7867468f8bb90: from storage DS-cd5de5bc-aae6-4c26-afd7-58c896927415 node DatanodeRegistration(127.0.0.1:45437, datanodeUuid=8050b55f-8f12-47d7-a080-9ddfbceb52d3, infoPort=46017, infoSecurePort=0, ipcPort=41107, storageInfo=lv=-57;cid=testClusterID;nsid=2007964022;c=1689444773408), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 18:12:57,243 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd917335d6ad3fd83: Processing first storage report for DS-bebda7c1-1962-40c0-a432-2ed88361e2ef from datanode eda77579-bb28-434f-b9ca-03ee57936a73 2023-07-15 18:12:57,243 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd917335d6ad3fd83: from storage DS-bebda7c1-1962-40c0-a432-2ed88361e2ef node DatanodeRegistration(127.0.0.1:39671, datanodeUuid=eda77579-bb28-434f-b9ca-03ee57936a73, infoPort=43803, infoSecurePort=0, ipcPort=46109, storageInfo=lv=-57;cid=testClusterID;nsid=2007964022;c=1689444773408), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 18:12:57,244 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd917335d6ad3fd83: Processing first storage report for DS-f2dcf18f-accc-49ee-8b11-5da78e359cd3 from datanode eda77579-bb28-434f-b9ca-03ee57936a73 2023-07-15 18:12:57,244 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd917335d6ad3fd83: from storage DS-f2dcf18f-accc-49ee-8b11-5da78e359cd3 node DatanodeRegistration(127.0.0.1:39671, datanodeUuid=eda77579-bb28-434f-b9ca-03ee57936a73, infoPort=43803, infoSecurePort=0, ipcPort=46109, storageInfo=lv=-57;cid=testClusterID;nsid=2007964022;c=1689444773408), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 18:12:57,606 DEBUG [Listener at localhost/46109] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0 2023-07-15 18:12:57,699 INFO [Listener at localhost/46109] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/zookeeper_0, clientPort=62148, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-15 18:12:57,720 INFO [Listener at localhost/46109] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62148 2023-07-15 18:12:57,732 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:12:57,735 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:12:58,477 INFO [Listener at localhost/46109] util.FSUtils(471): Created version file at hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 with version=8 2023-07-15 18:12:58,478 INFO [Listener at localhost/46109] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/hbase-staging 2023-07-15 18:12:58,488 DEBUG [Listener at localhost/46109] hbase.LocalHBaseCluster(134): Setting Master Port to random. 2023-07-15 18:12:58,489 DEBUG [Listener at localhost/46109] hbase.LocalHBaseCluster(141): Setting RegionServer Port to random. 2023-07-15 18:12:58,489 DEBUG [Listener at localhost/46109] hbase.LocalHBaseCluster(151): Setting RS InfoServer Port to random. 2023-07-15 18:12:58,489 DEBUG [Listener at localhost/46109] hbase.LocalHBaseCluster(159): Setting Master InfoServer Port to random. 2023-07-15 18:12:58,859 INFO [Listener at localhost/46109] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-15 18:12:59,479 INFO [Listener at localhost/46109] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:12:59,535 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:12:59,536 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:12:59,536 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:12:59,536 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:12:59,536 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:12:59,719 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:12:59,831 DEBUG [Listener at localhost/46109] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-15 18:12:59,945 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42385 2023-07-15 18:12:59,959 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:12:59,962 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:12:59,995 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42385 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:00,061 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:423850x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:00,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:42385-0x1016a3001d00000 connected 2023-07-15 18:13:00,104 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 18:13:00,105 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:00,110 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:00,128 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42385 2023-07-15 18:13:00,129 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42385 2023-07-15 18:13:00,132 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42385 2023-07-15 18:13:00,133 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42385 2023-07-15 18:13:00,133 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42385 2023-07-15 18:13:00,197 INFO [Listener at localhost/46109] log.Log(170): Logging initialized @9208ms to org.apache.hbase.thirdparty.org.eclipse.jetty.util.log.Slf4jLog 2023-07-15 18:13:00,426 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:00,427 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:00,428 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:00,430 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2023-07-15 18:13:00,431 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:00,431 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:00,436 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:00,536 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 41633 2023-07-15 18:13:00,544 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:00,651 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:00,654 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@1740f91f{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:00,655 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:00,655 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@15103edb{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:00,738 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:00,758 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:00,759 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:00,762 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 660000ms 2023-07-15 18:13:00,773 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:00,815 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@7c69fb5c{master,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/master/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/master} 2023-07-15 18:13:00,843 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@473902f7{HTTP/1.1, (http/1.1)}{0.0.0.0:41633} 2023-07-15 18:13:00,843 INFO [Listener at localhost/46109] server.Server(415): Started @9854ms 2023-07-15 18:13:00,849 INFO [Listener at localhost/46109] master.HMaster(444): hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117, hbase.cluster.distributed=false 2023-07-15 18:13:01,010 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:01,015 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,015 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,015 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:01,015 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,015 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:01,026 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:01,032 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43763 2023-07-15 18:13:01,036 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:01,047 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:01,048 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:01,052 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:01,055 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43763 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:01,064 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:437630x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:01,069 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:437630x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 18:13:01,082 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:01,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43763-0x1016a3001d00001 connected 2023-07-15 18:13:01,091 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:01,119 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43763 2023-07-15 18:13:01,119 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43763 2023-07-15 18:13:01,123 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43763 2023-07-15 18:13:01,126 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43763 2023-07-15 18:13:01,138 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43763 2023-07-15 18:13:01,142 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:01,143 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:01,143 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:01,152 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:01,153 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:01,153 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:01,153 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:01,159 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 40723 2023-07-15 18:13:01,159 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:01,182 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,182 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@639f3f91{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:01,183 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,183 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@4e584a37{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:01,201 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:01,203 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:01,204 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:01,204 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 660000ms 2023-07-15 18:13:01,205 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,210 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@32d72d2f{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:01,212 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@5b9eff03{HTTP/1.1, (http/1.1)}{0.0.0.0:40723} 2023-07-15 18:13:01,212 INFO [Listener at localhost/46109] server.Server(415): Started @10223ms 2023-07-15 18:13:01,226 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:01,226 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,226 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,227 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:01,227 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,227 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:01,227 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:01,233 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39349 2023-07-15 18:13:01,233 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:01,241 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:01,242 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:01,245 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:01,246 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39349 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:01,269 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:393490x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 18:13:01,271 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:393490x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:01,271 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:393490x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:01,274 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:393490x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:01,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39349-0x1016a3001d00002 connected 2023-07-15 18:13:01,331 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39349 2023-07-15 18:13:01,341 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39349 2023-07-15 18:13:01,356 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39349 2023-07-15 18:13:01,359 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39349 2023-07-15 18:13:01,362 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39349 2023-07-15 18:13:01,366 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:01,366 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:01,366 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:01,367 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:01,367 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:01,367 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:01,368 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:01,369 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 45977 2023-07-15 18:13:01,369 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:01,423 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,423 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@12839ff7{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:01,424 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,424 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@64c7beca{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:01,448 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:01,449 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:01,450 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:01,450 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 660000ms 2023-07-15 18:13:01,452 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,453 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@7a6bf43c{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:01,455 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@56035e5{HTTP/1.1, (http/1.1)}{0.0.0.0:45977} 2023-07-15 18:13:01,455 INFO [Listener at localhost/46109] server.Server(415): Started @10466ms 2023-07-15 18:13:01,475 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:01,475 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,475 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,475 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:01,476 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:01,476 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:01,476 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:01,480 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33055 2023-07-15 18:13:01,481 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:01,490 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:01,496 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:01,498 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:01,500 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33055 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:01,514 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:330550x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:01,516 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:330550x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 18:13:01,517 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:330550x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:01,518 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:330550x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:01,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33055-0x1016a3001d00003 connected 2023-07-15 18:13:01,535 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33055 2023-07-15 18:13:01,537 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33055 2023-07-15 18:13:01,555 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33055 2023-07-15 18:13:01,558 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33055 2023-07-15 18:13:01,565 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33055 2023-07-15 18:13:01,569 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:01,569 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:01,569 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:01,570 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:01,570 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:01,570 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:01,571 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:01,572 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 41271 2023-07-15 18:13:01,572 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:01,615 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,616 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@75642769{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:01,617 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,618 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@10470979{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:01,630 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:01,631 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:01,631 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:01,632 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 660000ms 2023-07-15 18:13:01,638 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:01,639 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@41f3e3cf{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:01,642 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@5922b370{HTTP/1.1, (http/1.1)}{0.0.0.0:41271} 2023-07-15 18:13:01,642 INFO [Listener at localhost/46109] server.Server(415): Started @10653ms 2023-07-15 18:13:01,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:01,750 INFO [master/jenkins-hbase4:0:becomeActiveMaster] server.AbstractConnector(333): Started ServerConnector@6a20a799{HTTP/1.1, (http/1.1)}{0.0.0.0:39663} 2023-07-15 18:13:01,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] server.Server(415): Started @10764ms 2023-07-15 18:13:01,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:01,777 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 18:13:01,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:01,848 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 18:13:01,848 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 18:13:01,849 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:01,851 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 18:13:01,852 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 18:13:01,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 18:13:01,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 18:13:01,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42385,1689444778672 from backup master directory 2023-07-15 18:13:01,868 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:01,868 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 18:13:01,870 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:01,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:01,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-15 18:13:01,895 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-15 18:13:02,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/hbase.id with ID: 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:02,290 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:02,314 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:02,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0689ffaf to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:02,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@610014ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:02,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 18:13:02,537 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-15 18:13:02,572 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-15 18:13:02,572 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-15 18:13:02,574 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:02,580 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:02,581 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:02,640 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store-tmp 2023-07-15 18:13:02,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:02,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 18:13:02,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:02,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:02,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 18:13:02,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:02,754 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:02,754 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 18:13:02,756 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/WALs/jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:02,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42385%2C1689444778672, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/WALs/jenkins-hbase4.apache.org,42385,1689444778672, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/oldWALs, maxLogs=10 2023-07-15 18:13:02,868 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:02,868 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:02,868 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:02,906 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:03,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/WALs/jenkins-hbase4.apache.org,42385,1689444778672/jenkins-hbase4.apache.org%2C42385%2C1689444778672.1689444782799 2023-07-15 18:13:03,077 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK], DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK]] 2023-07-15 18:13:03,078 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:03,078 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:03,084 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 18:13:03,086 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 18:13:03,283 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-15 18:13:03,295 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-15 18:13:03,358 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-15 18:13:03,379 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:03,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 18:13:03,388 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 18:13:03,433 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 18:13:03,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 18:13:03,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10383776960, jitterRate=-0.03293541073799133}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:03,452 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 18:13:03,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-15 18:13:03,501 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-15 18:13:03,501 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-15 18:13:03,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-15 18:13:03,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-15 18:13:03,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 61 msec 2023-07-15 18:13:03,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-15 18:13:03,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-15 18:13:03,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-15 18:13:03,634 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-15 18:13:03,642 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-15 18:13:03,651 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-15 18:13:03,657 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:03,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-15 18:13:03,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-15 18:13:03,684 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-15 18:13:03,692 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:03,692 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:03,693 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:03,695 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:03,695 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:03,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,42385,1689444778672, sessionid=0x1016a3001d00000, setting cluster-up flag (Was=false) 2023-07-15 18:13:03,723 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:03,734 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-15 18:13:03,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:03,744 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:03,755 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-15 18:13:03,761 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:03,766 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.hbase-snapshot/.tmp 2023-07-15 18:13:03,853 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:03,855 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:03,886 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:03,904 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:03,906 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:03,910 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:03,916 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:03,916 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:03,916 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:03,916 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:03,921 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:03,921 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:03,923 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:03,923 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:03,925 DEBUG [RS:0;jenkins-hbase4:43763] zookeeper.ReadOnlyZKClient(139): Connect 0x62c48004 to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:03,932 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:03,936 DEBUG [RS:2;jenkins-hbase4:33055] zookeeper.ReadOnlyZKClient(139): Connect 0x5db47ee4 to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:03,936 DEBUG [RS:1;jenkins-hbase4:39349] zookeeper.ReadOnlyZKClient(139): Connect 0x7e1b858b to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:03,945 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2930): Registered master coprocessor service: service=RSGroupAdminService 2023-07-15 18:13:03,976 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] rsgroup.RSGroupInfoManagerImpl(537): Refreshing in Offline mode. 2023-07-15 18:13:03,990 INFO [master/jenkins-hbase4:0:becomeActiveMaster] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint loaded, priority=536870911. 2023-07-15 18:13:03,990 INFO [master/jenkins-hbase4:0:becomeActiveMaster] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver loaded, priority=536870912. 2023-07-15 18:13:03,987 DEBUG [RS:0;jenkins-hbase4:43763] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c03722c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:03,999 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:04,000 DEBUG [RS:0;jenkins-hbase4:43763] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@683ceb74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:04,000 DEBUG [RS:2;jenkins-hbase4:33055] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53738b22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:04,000 DEBUG [RS:2;jenkins-hbase4:33055] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574c1594, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:04,002 DEBUG [RS:1;jenkins-hbase4:39349] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70dc6438, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:04,003 DEBUG [RS:1;jenkins-hbase4:39349] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31e16e03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:04,027 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39349 2023-07-15 18:13:04,029 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:33055 2023-07-15 18:13:04,032 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43763 2023-07-15 18:13:04,035 INFO [RS:1;jenkins-hbase4:39349] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:04,035 INFO [RS:0;jenkins-hbase4:43763] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:04,036 INFO [RS:0;jenkins-hbase4:43763] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:04,036 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:04,035 INFO [RS:2;jenkins-hbase4:33055] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:04,036 INFO [RS:2;jenkins-hbase4:33055] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:04,036 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:04,036 INFO [RS:1;jenkins-hbase4:39349] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:04,037 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:04,041 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:43763, startcode=1689444781009 2023-07-15 18:13:04,041 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:39349, startcode=1689444781225 2023-07-15 18:13:04,047 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:33055, startcode=1689444781474 2023-07-15 18:13:04,069 DEBUG [RS:2;jenkins-hbase4:33055] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:04,069 DEBUG [RS:0;jenkins-hbase4:43763] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:04,069 DEBUG [RS:1;jenkins-hbase4:39349] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:04,104 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-15 18:13:04,148 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47877, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:04,148 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52651, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:04,148 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35545, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:04,167 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 18:13:04,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-15 18:13:04,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-15 18:13:04,182 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 18:13:04,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-15 18:13:04,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-15 18:13:04,185 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 18:13:04,187 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 18:13:04,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 18:13:04,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 18:13:04,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 18:13:04,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-15 18:13:04,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:04,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,201 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689444814201 2023-07-15 18:13:04,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-15 18:13:04,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-15 18:13:04,210 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 18:13:04,211 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-15 18:13:04,215 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(2830): Master is not running yet 2023-07-15 18:13:04,215 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(2830): Master is not running yet 2023-07-15 18:13:04,215 WARN [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-15 18:13:04,215 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(2830): Master is not running yet 2023-07-15 18:13:04,215 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 18:13:04,215 WARN [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-15 18:13:04,215 WARN [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-15 18:13:04,227 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-15 18:13:04,228 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-15 18:13:04,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-15 18:13:04,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-15 18:13:04,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-15 18:13:04,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-15 18:13:04,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-15 18:13:04,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-15 18:13:04,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-15 18:13:04,259 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689444784250,5,FailOnTimeoutGroup] 2023-07-15 18:13:04,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689444784260,5,FailOnTimeoutGroup] 2023-07-15 18:13:04,260 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,260 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-15 18:13:04,262 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,262 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,344 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:33055, startcode=1689444781474 2023-07-15 18:13:04,344 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:43763, startcode=1689444781009 2023-07-15 18:13:04,344 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:39349, startcode=1689444781225 2023-07-15 18:13:04,350 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 18:13:04,351 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 18:13:04,351 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:04,352 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,354 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:04,355 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 1 2023-07-15 18:13:04,362 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,362 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:04,363 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 2 2023-07-15 18:13:04,363 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:04,363 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:04,364 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:04,364 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,364 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:04,364 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 3 2023-07-15 18:13:04,365 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:04,365 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:04,365 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:04,366 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:04,366 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:04,366 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:04,379 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:04,390 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:04,391 DEBUG [RS:2;jenkins-hbase4:33055] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,391 WARN [RS:2;jenkins-hbase4:33055] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:04,392 DEBUG [RS:1;jenkins-hbase4:39349] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,392 DEBUG [RS:0;jenkins-hbase4:43763] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,392 WARN [RS:1;jenkins-hbase4:39349] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:04,392 INFO [RS:2;jenkins-hbase4:33055] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:04,392 INFO [RS:1;jenkins-hbase4:39349] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:04,392 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39349,1689444781225] 2023-07-15 18:13:04,393 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,392 WARN [RS:0;jenkins-hbase4:43763] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:04,393 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43763,1689444781009] 2023-07-15 18:13:04,393 INFO [RS:0;jenkins-hbase4:43763] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:04,393 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,393 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,393 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33055,1689444781474] 2023-07-15 18:13:04,395 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 18:13:04,401 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:04,402 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 18:13:04,403 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:04,403 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 18:13:04,406 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:04,407 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 18:13:04,408 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:04,409 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 18:13:04,411 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:04,412 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 18:13:04,413 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:04,414 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:04,416 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:04,420 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 18:13:04,421 DEBUG [RS:1;jenkins-hbase4:39349] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,421 DEBUG [RS:2;jenkins-hbase4:33055] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,422 DEBUG [RS:0;jenkins-hbase4:43763] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,423 DEBUG [RS:1;jenkins-hbase4:39349] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,423 DEBUG [RS:2;jenkins-hbase4:33055] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,423 DEBUG [RS:0;jenkins-hbase4:43763] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,424 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 18:13:04,424 DEBUG [RS:1;jenkins-hbase4:39349] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,424 DEBUG [RS:2;jenkins-hbase4:33055] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,424 DEBUG [RS:0;jenkins-hbase4:43763] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,430 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 18:13:04,431 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=11938448000, jitterRate=0.11185461282730103}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 18:13:04,431 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 18:13:04,431 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 18:13:04,432 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 18:13:04,432 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 18:13:04,432 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 18:13:04,432 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 18:13:04,433 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 18:13:04,434 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 18:13:04,441 DEBUG [RS:2;jenkins-hbase4:33055] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:04,441 DEBUG [RS:0;jenkins-hbase4:43763] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:04,441 DEBUG [RS:1;jenkins-hbase4:39349] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:04,445 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 18:13:04,445 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-15 18:13:04,456 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 18:13:04,457 INFO [RS:1;jenkins-hbase4:39349] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:04,457 INFO [RS:2;jenkins-hbase4:33055] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:04,458 INFO [RS:0;jenkins-hbase4:43763] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:04,473 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 18:13:04,478 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-15 18:13:04,490 INFO [RS:2;jenkins-hbase4:33055] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:04,490 INFO [RS:1;jenkins-hbase4:39349] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:04,490 INFO [RS:0;jenkins-hbase4:43763] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:04,495 INFO [RS:1;jenkins-hbase4:39349] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:04,495 INFO [RS:2;jenkins-hbase4:33055] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:04,495 INFO [RS:0;jenkins-hbase4:43763] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:04,496 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,496 INFO [RS:1;jenkins-hbase4:39349] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,497 INFO [RS:0;jenkins-hbase4:43763] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,498 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:04,498 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:04,498 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:04,506 INFO [RS:0;jenkins-hbase4:43763] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,506 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,506 INFO [RS:1;jenkins-hbase4:39349] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,507 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,507 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:04,508 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,508 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:04,508 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:04,509 DEBUG [RS:0;jenkins-hbase4:43763] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,509 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,510 DEBUG [RS:2;jenkins-hbase4:33055] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,510 DEBUG [RS:1;jenkins-hbase4:39349] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:04,511 INFO [RS:0;jenkins-hbase4:43763] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,512 INFO [RS:0;jenkins-hbase4:43763] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,512 INFO [RS:0;jenkins-hbase4:43763] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,513 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,513 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,513 INFO [RS:1;jenkins-hbase4:39349] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,513 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,513 INFO [RS:1;jenkins-hbase4:39349] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,513 INFO [RS:1;jenkins-hbase4:39349] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,531 INFO [RS:0;jenkins-hbase4:43763] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:04,531 INFO [RS:2;jenkins-hbase4:33055] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:04,531 INFO [RS:1;jenkins-hbase4:39349] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:04,535 INFO [RS:0;jenkins-hbase4:43763] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43763,1689444781009-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,535 INFO [RS:1;jenkins-hbase4:39349] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39349,1689444781225-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,535 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33055,1689444781474-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:04,554 INFO [RS:0;jenkins-hbase4:43763] regionserver.Replication(203): jenkins-hbase4.apache.org,43763,1689444781009 started 2023-07-15 18:13:04,554 INFO [RS:1;jenkins-hbase4:39349] regionserver.Replication(203): jenkins-hbase4.apache.org,39349,1689444781225 started 2023-07-15 18:13:04,554 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43763,1689444781009, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43763, sessionid=0x1016a3001d00001 2023-07-15 18:13:04,554 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39349,1689444781225, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39349, sessionid=0x1016a3001d00002 2023-07-15 18:13:04,554 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:04,554 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:04,554 DEBUG [RS:0;jenkins-hbase4:43763] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,554 DEBUG [RS:1;jenkins-hbase4:39349] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,555 DEBUG [RS:0;jenkins-hbase4:43763] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43763,1689444781009' 2023-07-15 18:13:04,555 DEBUG [RS:1;jenkins-hbase4:39349] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39349,1689444781225' 2023-07-15 18:13:04,556 DEBUG [RS:1;jenkins-hbase4:39349] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:04,556 DEBUG [RS:0;jenkins-hbase4:43763] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:04,557 DEBUG [RS:0;jenkins-hbase4:43763] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:04,557 DEBUG [RS:1;jenkins-hbase4:39349] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:04,557 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:04,557 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:04,557 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:04,557 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:04,558 DEBUG [RS:0;jenkins-hbase4:43763] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,558 DEBUG [RS:1;jenkins-hbase4:39349] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:04,558 DEBUG [RS:0;jenkins-hbase4:43763] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43763,1689444781009' 2023-07-15 18:13:04,558 DEBUG [RS:0;jenkins-hbase4:43763] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:04,558 DEBUG [RS:1;jenkins-hbase4:39349] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39349,1689444781225' 2023-07-15 18:13:04,558 DEBUG [RS:1;jenkins-hbase4:39349] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:04,559 DEBUG [RS:0;jenkins-hbase4:43763] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:04,559 DEBUG [RS:1;jenkins-hbase4:39349] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:04,559 INFO [RS:2;jenkins-hbase4:33055] regionserver.Replication(203): jenkins-hbase4.apache.org,33055,1689444781474 started 2023-07-15 18:13:04,559 DEBUG [RS:0;jenkins-hbase4:43763] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:04,559 DEBUG [RS:1;jenkins-hbase4:39349] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:04,559 INFO [RS:0;jenkins-hbase4:43763] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:04,559 INFO [RS:1;jenkins-hbase4:39349] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:04,559 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,33055,1689444781474, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33055, sessionid=0x1016a3001d00003 2023-07-15 18:13:04,560 INFO [RS:1;jenkins-hbase4:39349] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:04,560 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:04,560 DEBUG [RS:2;jenkins-hbase4:33055] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,560 INFO [RS:0;jenkins-hbase4:43763] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:04,560 DEBUG [RS:2;jenkins-hbase4:33055] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33055,1689444781474' 2023-07-15 18:13:04,561 DEBUG [RS:2;jenkins-hbase4:33055] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:04,561 DEBUG [RS:2;jenkins-hbase4:33055] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:04,561 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:04,561 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:04,561 DEBUG [RS:2;jenkins-hbase4:33055] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:04,561 DEBUG [RS:2;jenkins-hbase4:33055] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33055,1689444781474' 2023-07-15 18:13:04,562 DEBUG [RS:2;jenkins-hbase4:33055] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:04,562 DEBUG [RS:2;jenkins-hbase4:33055] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:04,562 DEBUG [RS:2;jenkins-hbase4:33055] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:04,562 INFO [RS:2;jenkins-hbase4:33055] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:04,562 INFO [RS:2;jenkins-hbase4:33055] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:04,630 DEBUG [jenkins-hbase4:42385] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-07-15 18:13:04,654 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 18:13:04,656 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 18:13:04,656 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 18:13:04,656 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-07-15 18:13:04,656 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 18:13:04,662 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43763,1689444781009, state=OPENING 2023-07-15 18:13:04,677 INFO [RS:0;jenkins-hbase4:43763] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43763%2C1689444781009, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:04,677 INFO [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33055%2C1689444781474, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,33055,1689444781474, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:04,683 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-15 18:13:04,683 INFO [RS:1;jenkins-hbase4:39349] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39349%2C1689444781225, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:04,685 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:04,686 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:04,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43763,1689444781009}] 2023-07-15 18:13:04,716 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:04,717 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:04,719 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:04,720 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:04,720 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:04,720 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:04,721 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:04,721 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:04,722 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:04,732 INFO [RS:1;jenkins-hbase4:39349] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 2023-07-15 18:13:04,733 DEBUG [RS:1;jenkins-hbase4:39349] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:04,735 INFO [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,33055,1689444781474/jenkins-hbase4.apache.org%2C33055%2C1689444781474.1689444784682 2023-07-15 18:13:04,735 INFO [RS:0;jenkins-hbase4:43763] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 2023-07-15 18:13:04,735 DEBUG [RS:0;jenkins-hbase4:43763] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:04,736 DEBUG [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:04,781 WARN [ReadOnlyZKClient-127.0.0.1:62148@0x0689ffaf] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 18:13:04,813 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:04,821 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:04,822 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43763] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Get size: 88 connection: 172.31.14.131:47438 deadline: 1689444844822, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,901 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:04,906 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 18:13:04,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 18:13:04,927 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 18:13:04,927 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:04,933 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta, suffix=.meta, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:04,956 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:04,956 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:04,957 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:04,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta 2023-07-15 18:13:04,971 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK]] 2023-07-15 18:13:04,971 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:04,973 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:04,976 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 18:13:04,978 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 18:13:04,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 18:13:04,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:04,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 18:13:04,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 18:13:04,987 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 18:13:04,989 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:04,989 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:04,990 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 18:13:04,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:04,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 18:13:04,993 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:04,993 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:04,993 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 18:13:04,994 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:04,995 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 18:13:04,996 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:04,996 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:04,997 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 18:13:04,997 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:04,999 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:05,002 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:05,006 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 18:13:05,009 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 18:13:05,011 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10720372480, jitterRate=-0.001587510108947754}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 18:13:05,011 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 18:13:05,023 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689444784897 2023-07-15 18:13:05,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 18:13:05,047 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 18:13:05,047 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43763,1689444781009, state=OPEN 2023-07-15 18:13:05,052 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 18:13:05,053 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:05,057 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-15 18:13:05,058 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43763,1689444781009 in 362 msec 2023-07-15 18:13:05,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-15 18:13:05,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 604 msec 2023-07-15 18:13:05,075 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0690 sec 2023-07-15 18:13:05,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689444785077, completionTime=-1 2023-07-15 18:13:05,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-07-15 18:13:05,077 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-15 18:13:05,141 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-07-15 18:13:05,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689444845142 2023-07-15 18:13:05,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689444905142 2023-07-15 18:13:05,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 64 msec 2023-07-15 18:13:05,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42385,1689444778672-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:05,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42385,1689444778672-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:05,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42385,1689444778672-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:05,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42385, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:05,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:05,173 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-15 18:13:05,192 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-15 18:13:05,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 18:13:05,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-15 18:13:05,210 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 18:13:05,213 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 18:13:05,232 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,235 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 empty. 2023-07-15 18:13:05,236 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,236 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-15 18:13:05,284 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-15 18:13:05,287 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 83c5639bc741d79ce2b4c69e5abc7617, NAME => 'hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp 2023-07-15 18:13:05,308 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:05,308 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 83c5639bc741d79ce2b4c69e5abc7617, disabling compactions & flushes 2023-07-15 18:13:05,308 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,308 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,308 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. after waiting 0 ms 2023-07-15 18:13:05,308 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,308 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,308 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:05,314 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 18:13:05,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444785318"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444785318"}]},"ts":"1689444785318"} 2023-07-15 18:13:05,343 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] master.HMaster(2148): Client=null/null create 'hbase:rsgroup', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|', METADATA => {'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}}, {NAME => 'm', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 18:13:05,346 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] procedure2.ProcedureExecutor(1029): Stored pid=5, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:rsgroup 2023-07-15 18:13:05,349 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=5, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:rsgroup execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 18:13:05,351 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=5, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:rsgroup execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 18:13:05,355 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,356 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 empty. 2023-07-15 18:13:05,357 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,357 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:rsgroup regions 2023-07-15 18:13:05,374 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 18:13:05,379 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 18:13:05,388 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/hbase/rsgroup/.tabledesc/.tableinfo.0000000001 2023-07-15 18:13:05,389 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444785379"}]},"ts":"1689444785379"} 2023-07-15 18:13:05,390 INFO [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(7675): creating {ENCODED => 994da785cf935856bc388ebe43136633, NAME => 'hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:rsgroup', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|', METADATA => {'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}}, {NAME => 'm', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp 2023-07-15 18:13:05,393 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-15 18:13:05,406 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 18:13:05,407 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 18:13:05,407 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 18:13:05,407 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-07-15 18:13:05,407 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 18:13:05,410 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN}] 2023-07-15 18:13:05,413 DEBUG [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(866): Instantiated hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:05,414 DEBUG [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1604): Closing 994da785cf935856bc388ebe43136633, disabling compactions & flushes 2023-07-15 18:13:05,414 INFO [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1626): Closing region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,414 DEBUG [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,414 DEBUG [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. after waiting 0 ms 2023-07-15 18:13:05,414 DEBUG [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,414 INFO [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1838): Closed hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,414 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=6, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN 2023-07-15 18:13:05,414 DEBUG [RegionOpenAndInit-hbase:rsgroup-pool-0] regionserver.HRegion(1558): Region close journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:05,416 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=6, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39349,1689444781225; forceNewPlan=false, retain=false 2023-07-15 18:13:05,419 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=5, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:rsgroup execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 18:13:05,420 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444785420"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444785420"}]},"ts":"1689444785420"} 2023-07-15 18:13:05,424 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 18:13:05,425 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=5, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:rsgroup execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 18:13:05,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:rsgroup","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444785425"}]},"ts":"1689444785425"} 2023-07-15 18:13:05,431 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:rsgroup, state=ENABLING in hbase:meta 2023-07-15 18:13:05,435 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 18:13:05,435 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 18:13:05,435 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 18:13:05,435 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-07-15 18:13:05,436 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 18:13:05,436 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=7, ppid=5, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN}] 2023-07-15 18:13:05,439 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=7, ppid=5, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN 2023-07-15 18:13:05,440 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=7, ppid=5, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33055,1689444781474; forceNewPlan=false, retain=false 2023-07-15 18:13:05,441 INFO [jenkins-hbase4:42385] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-07-15 18:13:05,443 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=6 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:05,443 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=7 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:05,443 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444785443"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444785443"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444785443"}]},"ts":"1689444785443"} 2023-07-15 18:13:05,443 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444785442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444785442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444785442"}]},"ts":"1689444785442"} 2023-07-15 18:13:05,447 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=8, ppid=6, state=RUNNABLE; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,39349,1689444781225}] 2023-07-15 18:13:05,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=9, ppid=7, state=RUNNABLE; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:05,602 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:05,602 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 18:13:05,603 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:05,604 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 18:13:05,605 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53716, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 18:13:05,607 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 18:13:05,612 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,612 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 83c5639bc741d79ce2b4c69e5abc7617, NAME => 'hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:05,613 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,613 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 994da785cf935856bc388ebe43136633, NAME => 'hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:05,613 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:05,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:05,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. service=MultiRowMutationService 2023-07-15 18:13:05,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,615 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:rsgroup successfully. 2023-07-15 18:13:05,615 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table rsgroup 994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,615 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:05,615 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,615 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,617 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,618 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family m of region 994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,621 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:05,621 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:05,621 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:05,621 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:05,621 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 83c5639bc741d79ce2b4c69e5abc7617 columnFamilyName info 2023-07-15 18:13:05,621 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 994da785cf935856bc388ebe43136633 columnFamilyName m 2023-07-15 18:13:05,622 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(310): Store=83c5639bc741d79ce2b4c69e5abc7617/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:05,623 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(310): Store=994da785cf935856bc388ebe43136633/m, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:05,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,630 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:05,630 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:05,635 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 18:13:05,636 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 83c5639bc741d79ce2b4c69e5abc7617; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=11755569600, jitterRate=0.09482273459434509}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:05,636 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 18:13:05,636 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:05,637 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 994da785cf935856bc388ebe43136633; next sequenceid=2; org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy@c5297fd, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:05,637 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:05,639 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617., pid=8, masterSystemTime=1689444785602 2023-07-15 18:13:05,645 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633., pid=9, masterSystemTime=1689444785603 2023-07-15 18:13:05,647 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,647 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:05,650 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=6 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:05,650 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444785649"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444785649"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444785649"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444785649"}]},"ts":"1689444785649"} 2023-07-15 18:13:05,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,652 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:05,652 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=7 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:05,653 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444785652"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444785652"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444785652"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444785652"}]},"ts":"1689444785652"} 2023-07-15 18:13:05,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=8, resume processing ppid=6 2023-07-15 18:13:05,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, ppid=6, state=SUCCESS; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,39349,1689444781225 in 207 msec 2023-07-15 18:13:05,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=9, resume processing ppid=7 2023-07-15 18:13:05,663 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, ppid=7, state=SUCCESS; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474 in 208 msec 2023-07-15 18:13:05,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=4 2023-07-15 18:13:05,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN in 249 msec 2023-07-15 18:13:05,667 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 18:13:05,668 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444785667"}]},"ts":"1689444785667"} 2023-07-15 18:13:05,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=7, resume processing ppid=5 2023-07-15 18:13:05,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, ppid=5, state=SUCCESS; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN in 227 msec 2023-07-15 18:13:05,671 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=5, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:rsgroup execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 18:13:05,672 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:rsgroup","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444785671"}]},"ts":"1689444785671"} 2023-07-15 18:13:05,675 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-15 18:13:05,677 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:rsgroup, state=ENABLED in hbase:meta 2023-07-15 18:13:05,680 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 18:13:05,686 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=5, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:rsgroup execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 18:13:05,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 486 msec 2023-07-15 18:13:05,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, state=SUCCESS; CreateTableProcedure table=hbase:rsgroup in 343 msec 2023-07-15 18:13:05,710 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-15 18:13:05,711 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-15 18:13:05,712 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:05,740 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:05,759 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:05,764 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:05,770 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:05,774 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker(840): RSGroup table=hbase:rsgroup is online, refreshing cached information 2023-07-15 18:13:05,774 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(534): Refreshing in Online mode. 2023-07-15 18:13:05,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=10, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-15 18:13:05,811 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 18:13:05,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 53 msec 2023-07-15 18:13:05,829 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=11, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-15 18:13:05,856 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 18:13:05,864 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 32 msec 2023-07-15 18:13:05,875 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-15 18:13:05,877 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:05,877 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:05,879 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-15 18:13:05,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.009sec 2023-07-15 18:13:05,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-15 18:13:05,884 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 2 2023-07-15 18:13:05,885 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-15 18:13:05,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-15 18:13:05,889 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42385,1689444778672-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-15 18:13:05,890 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42385,1689444778672-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-15 18:13:05,896 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker(823): GroupBasedLoadBalancer is now online 2023-07-15 18:13:05,906 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-15 18:13:05,986 DEBUG [Listener at localhost/46109] zookeeper.ReadOnlyZKClient(139): Connect 0x7075917a to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:05,993 DEBUG [Listener at localhost/46109] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b1c4f85, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:06,013 DEBUG [hconnection-0x2af4e333-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:06,028 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:06,042 INFO [Listener at localhost/46109] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:06,044 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:06,057 DEBUG [Listener at localhost/46109] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-15 18:13:06,061 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-15 18:13:06,081 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-07-15 18:13:06,081 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:06,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-07-15 18:13:06,088 DEBUG [Listener at localhost/46109] zookeeper.ReadOnlyZKClient(139): Connect 0x5e275124 to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:06,096 DEBUG [Listener at localhost/46109] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60eb43e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:06,096 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=VerifyingRSGroupAdminClient connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:06,104 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-07-15 18:13:06,106 INFO [Listener at localhost/46109] master.MasterRpcServices(492): Client=null/null set balanceSwitch=true 2023-07-15 18:13:06,106 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): VerifyingRSGroupAdminClient0x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:06,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): VerifyingRSGroupAdminClient-0x1016a3001d0000a connected 2023-07-15 18:13:06,151 INFO [Listener at localhost/46109] hbase.ResourceChecker(147): before: rsgroup.TestRSGroupsFallback#testFallback Thread=425, OpenFileDescriptor=679, MaxFileDescriptor=60000, SystemLoadAverage=776, ProcessCount=183, AvailableMemoryMB=1443 2023-07-15 18:13:06,155 INFO [Listener at localhost/46109] rsgroup.TestRSGroupsBase(132): testFallback 2023-07-15 18:13:06,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:06,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:06,258 INFO [Listener at localhost/46109] rsgroup.TestRSGroupsBase(152): Restoring servers: 1 2023-07-15 18:13:06,275 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:06,275 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:06,276 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:06,276 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:06,276 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:06,276 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:06,276 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:06,281 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42063 2023-07-15 18:13:06,282 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:06,288 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:06,290 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:06,296 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:06,299 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42063 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:06,308 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:420630x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:06,310 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:420630x0, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 18:13:06,335 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 18:13:06,336 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:06,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42063-0x1016a3001d0000b connected 2023-07-15 18:13:06,346 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42063 2023-07-15 18:13:06,348 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42063 2023-07-15 18:13:06,350 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42063 2023-07-15 18:13:06,351 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42063 2023-07-15 18:13:06,354 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42063 2023-07-15 18:13:06,357 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:06,357 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:06,357 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:06,358 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:06,358 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:06,358 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:06,358 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:06,359 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 39715 2023-07-15 18:13:06,359 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:06,364 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:06,365 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@4deaba7a{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:06,365 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:06,365 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@745540d2{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:06,376 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:06,377 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:06,377 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:06,378 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 600000ms 2023-07-15 18:13:06,379 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:06,380 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@407fb65b{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:06,381 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@26f2f4a9{HTTP/1.1, (http/1.1)}{0.0.0.0:39715} 2023-07-15 18:13:06,382 INFO [Listener at localhost/46109] server.Server(415): Started @15392ms 2023-07-15 18:13:06,387 INFO [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:06,387 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:06,391 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:06,391 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:06,393 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:06,395 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ReadOnlyZKClient(139): Connect 0x5abaedce to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:06,409 DEBUG [RS:3;jenkins-hbase4:42063] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@547d08e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:06,409 DEBUG [RS:3;jenkins-hbase4:42063] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@504286be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:06,422 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:42063 2023-07-15 18:13:06,422 INFO [RS:3;jenkins-hbase4:42063] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:06,423 INFO [RS:3;jenkins-hbase4:42063] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:06,423 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:06,424 INFO [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:42063, startcode=1689444786274 2023-07-15 18:13:06,424 DEBUG [RS:3;jenkins-hbase4:42063] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:06,439 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43411, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:06,440 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,440 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:06,441 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:06,441 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:06,441 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:06,443 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:06,443 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:06,444 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:06,444 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:06,448 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42063,1689444786274] 2023-07-15 18:13:06,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:06,449 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,449 WARN [RS:3;jenkins-hbase4:42063] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:06,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:06,449 INFO [RS:3;jenkins-hbase4:42063] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:06,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:06,450 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:06,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:06,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:06,451 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:06,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:06,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:06,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:06,458 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 2 2023-07-15 18:13:06,461 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 4 2023-07-15 18:13:06,467 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:06,468 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,468 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:06,469 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:06,471 DEBUG [RS:3;jenkins-hbase4:42063] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:06,472 INFO [RS:3;jenkins-hbase4:42063] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:06,476 INFO [RS:3;jenkins-hbase4:42063] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:06,492 INFO [RS:3;jenkins-hbase4:42063] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:06,492 INFO [RS:3;jenkins-hbase4:42063] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:06,498 INFO [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:06,501 INFO [RS:3;jenkins-hbase4:42063] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,501 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,502 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,502 DEBUG [RS:3;jenkins-hbase4:42063] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:06,506 INFO [RS:3;jenkins-hbase4:42063] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:06,507 INFO [RS:3;jenkins-hbase4:42063] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:06,507 INFO [RS:3;jenkins-hbase4:42063] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:06,525 INFO [RS:3;jenkins-hbase4:42063] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:06,525 INFO [RS:3;jenkins-hbase4:42063] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42063,1689444786274-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:06,543 INFO [RS:3;jenkins-hbase4:42063] regionserver.Replication(203): jenkins-hbase4.apache.org,42063,1689444786274 started 2023-07-15 18:13:06,543 INFO [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,42063,1689444786274, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42063, sessionid=0x1016a3001d0000b 2023-07-15 18:13:06,543 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:06,543 DEBUG [RS:3;jenkins-hbase4:42063] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,543 DEBUG [RS:3;jenkins-hbase4:42063] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42063,1689444786274' 2023-07-15 18:13:06,543 DEBUG [RS:3;jenkins-hbase4:42063] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:06,544 DEBUG [RS:3;jenkins-hbase4:42063] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:06,545 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:06,545 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:06,545 DEBUG [RS:3;jenkins-hbase4:42063] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:06,545 DEBUG [RS:3;jenkins-hbase4:42063] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42063,1689444786274' 2023-07-15 18:13:06,545 DEBUG [RS:3;jenkins-hbase4:42063] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:06,545 DEBUG [RS:3;jenkins-hbase4:42063] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:06,547 DEBUG [RS:3;jenkins-hbase4:42063] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:06,547 INFO [RS:3;jenkins-hbase4:42063] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:06,547 INFO [RS:3;jenkins-hbase4:42063] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:06,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(252): Client=jenkins//172.31.14.131 add rsgroup master 2023-07-15 18:13:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:06,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:06,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 4 2023-07-15 18:13:06,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.AddRSGroup 2023-07-15 18:13:06,567 DEBUG [hconnection-0x5f48d716-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:06,575 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:06,585 DEBUG [hconnection-0x5f48d716-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:06,589 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:06,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:06,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:06,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:42385] to rsgroup master 2023-07-15 18:13:06,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:408) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:213) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16193) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:900) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 18:13:06,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] ipc.CallRunner(144): callId: 20 service: MasterService methodName: ExecMasterService size: 118 connection: 172.31.14.131:40862 deadline: 1689445986605, exception=org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. 2023-07-15 18:13:06,607 WARN [Listener at localhost/46109] rsgroup.TestRSGroupsBase(163): Got this on setup, FYI org.apache.hadoop.hbase.constraint.ConstraintException: org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:408) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:213) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16193) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:900) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101) at org.apache.hadoop.hbase.client.HBaseAdmin$74.callExecService(HBaseAdmin.java:2985) at org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:62) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService$BlockingStub.moveServers(RSGroupAdminProtos.java:16658) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient.moveServers(RSGroupAdminClient.java:108) at org.apache.hadoop.hbase.rsgroup.VerifyingRSGroupAdminClient.moveServers(VerifyingRSGroupAdminClient.java:77) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase.tearDownAfterMethod(TestRSGroupsBase.java:161) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase.setUpBeforeMethod(TestRSGroupsBase.java:136) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsFallback.beforeMethod(TestRSGroupsFallback.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.constraint.ConstraintException): org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:408) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:213) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16193) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:900) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 18:13:06,609 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:06,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:06,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:06,615 INFO [Listener at localhost/46109] rsgroup.TestRSGroupsBase$2(169): Waiting for cleanup to finish [Name:default, Servers:[jenkins-hbase4.apache.org:33055, jenkins-hbase4.apache.org:39349, jenkins-hbase4.apache.org:42063, jenkins-hbase4.apache.org:43763], Tables:[hbase:meta, hbase:namespace, hbase:rsgroup], Configurations:{}, Name:master, Servers:[], Tables:[], Configurations:{}] 2023-07-15 18:13:06,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:06,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:06,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:06,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:06,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(252): Client=jenkins//172.31.14.131 add rsgroup fallback 2023-07-15 18:13:06,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:06,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:06,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:06,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 6 2023-07-15 18:13:06,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.AddRSGroup 2023-07-15 18:13:06,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:06,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:06,651 INFO [RS:3;jenkins-hbase4:42063] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42063%2C1689444786274, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:06,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:33055] to rsgroup fallback 2023-07-15 18:13:06,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:06,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:06,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 6 2023-07-15 18:13:06,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(238): Moving server region 994da785cf935856bc388ebe43136633, which do not belong to RSGroup fallback 2023-07-15 18:13:06,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE 2023-07-15 18:13:06,668 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE 2023-07-15 18:13:06,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(286): Moving 1 region(s) to group default, current retry=0 2023-07-15 18:13:06,670 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:06,670 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444786670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444786670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444786670"}]},"ts":"1689444786670"} 2023-07-15 18:13:06,673 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE; CloseRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:06,709 DEBUG [RS-EventLoopGroup-7-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:06,710 DEBUG [RS-EventLoopGroup-7-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:06,711 DEBUG [RS-EventLoopGroup-7-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:06,719 INFO [RS:3;jenkins-hbase4:42063] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 2023-07-15 18:13:06,722 DEBUG [RS:3;jenkins-hbase4:42063] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:06,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 994da785cf935856bc388ebe43136633 2023-07-15 18:13:06,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 994da785cf935856bc388ebe43136633, disabling compactions & flushes 2023-07-15 18:13:06,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:06,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:06,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. after waiting 0 ms 2023-07-15 18:13:06,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:06,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 994da785cf935856bc388ebe43136633 1/1 column families, dataSize=1.24 KB heapSize=2.22 KB 2023-07-15 18:13:07,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.24 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/1cca1b552a7046428556b030adf432e1 2023-07-15 18:13:07,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/1cca1b552a7046428556b030adf432e1 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1 2023-07-15 18:13:07,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1, entries=3, sequenceid=9, filesize=5.0 K 2023-07-15 18:13:07,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.24 KB/1270, heapSize ~2.20 KB/2256, currentSize=0 B/0 for 994da785cf935856bc388ebe43136633 in 274ms, sequenceid=9, compaction requested=false 2023-07-15 18:13:07,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:rsgroup' 2023-07-15 18:13:07,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-07-15 18:13:07,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 18:13:07,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:07,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:07,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3513): Adding 994da785cf935856bc388ebe43136633 move to jenkins-hbase4.apache.org,42063,1689444786274 record at close sequenceid=9 2023-07-15 18:13:07,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,151 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=CLOSED 2023-07-15 18:13:07,151 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444787151"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444787151"}]},"ts":"1689444787151"} 2023-07-15 18:13:07,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-15 18:13:07,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; CloseRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474 in 481 msec 2023-07-15 18:13:07,160 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42063,1689444786274; forceNewPlan=false, retain=false 2023-07-15 18:13:07,311 INFO [jenkins-hbase4:42385] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 18:13:07,311 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:07,311 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444787311"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444787311"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444787311"}]},"ts":"1689444787311"} 2023-07-15 18:13:07,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=12, state=RUNNABLE; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,42063,1689444786274}] 2023-07-15 18:13:07,469 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:07,469 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 18:13:07,472 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 18:13:07,478 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:07,478 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 994da785cf935856bc388ebe43136633, NAME => 'hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:07,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:07,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. service=MultiRowMutationService 2023-07-15 18:13:07,479 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:rsgroup successfully. 2023-07-15 18:13:07,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table rsgroup 994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:07,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,481 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family m of region 994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,483 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:07,483 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:07,484 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 994da785cf935856bc388ebe43136633 columnFamilyName m 2023-07-15 18:13:07,512 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1 2023-07-15 18:13:07,513 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(310): Store=994da785cf935856bc388ebe43136633/m, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:07,515 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,518 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,523 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:07,524 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 994da785cf935856bc388ebe43136633; next sequenceid=13; org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy@67b211c8, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:07,524 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:07,526 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633., pid=14, masterSystemTime=1689444787469 2023-07-15 18:13:07,531 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:07,532 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:07,533 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPEN, openSeqNum=13, regionLocation=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:07,533 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444787533"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444787533"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444787533"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444787533"}]},"ts":"1689444787533"} 2023-07-15 18:13:07,541 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=12 2023-07-15 18:13:07,541 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=12, state=SUCCESS; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,42063,1689444786274 in 222 msec 2023-07-15 18:13:07,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE in 875 msec 2023-07-15 18:13:07,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure.ProcedureSyncWait(216): waitFor pid=12 2023-07-15 18:13:07,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(261): All regions from [jenkins-hbase4.apache.org,33055,1689444781474] are moved back to default 2023-07-15 18:13:07,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(438): Move servers done: default => fallback 2023-07-15 18:13:07,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveServers 2023-07-15 18:13:07,674 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33055] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 136 connection: 172.31.14.131:33804 deadline: 1689444847673, exception=org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=jenkins-hbase4.apache.org port=42063 startCode=1689444786274. As of locationSeqNum=9. 2023-07-15 18:13:07,781 DEBUG [hconnection-0x5f48d716-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:07,788 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:07,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:07,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:07,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=fallback 2023-07-15 18:13:07,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:07,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:07,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:07,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(252): Client=jenkins//172.31.14.131 add rsgroup Group_testFallback_1624499712 2023-07-15 18:13:07,827 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33055] ipc.CallRunner(144): callId: 43 service: ClientService methodName: ExecService size: 598 connection: 172.31.14.131:33790 deadline: 1689444847827, exception=org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=jenkins-hbase4.apache.org port=42063 startCode=1689444786274. As of locationSeqNum=9. 2023-07-15 18:13:07,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:07,936 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:07,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:07,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:07,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:07,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:07,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:07,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.AddRSGroup 2023-07-15 18:13:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:07,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:39349] to rsgroup Group_testFallback_1624499712 2023-07-15 18:13:07,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:07,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:07,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:07,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:07,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:07,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(238): Moving server region 83c5639bc741d79ce2b4c69e5abc7617, which do not belong to RSGroup Group_testFallback_1624499712 2023-07-15 18:13:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 18:13:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 18:13:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 18:13:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-07-15 18:13:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 18:13:08,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE 2023-07-15 18:13:08,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(286): Moving 1 region(s) to group default, current retry=0 2023-07-15 18:13:08,002 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE 2023-07-15 18:13:08,003 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:08,004 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444788003"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444788003"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444788003"}]},"ts":"1689444788003"} 2023-07-15 18:13:08,007 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; CloseRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,39349,1689444781225}] 2023-07-15 18:13:08,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 83c5639bc741d79ce2b4c69e5abc7617, disabling compactions & flushes 2023-07-15 18:13:08,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. after waiting 0 ms 2023-07-15 18:13:08,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 83c5639bc741d79ce2b4c69e5abc7617 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-15 18:13:08,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/.tmp/info/cdaf017d393d46f59089b5495bf62ca2 2023-07-15 18:13:08,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/.tmp/info/cdaf017d393d46f59089b5495bf62ca2 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info/cdaf017d393d46f59089b5495bf62ca2 2023-07-15 18:13:08,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info/cdaf017d393d46f59089b5495bf62ca2, entries=2, sequenceid=6, filesize=4.8 K 2023-07-15 18:13:08,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 83c5639bc741d79ce2b4c69e5abc7617 in 70ms, sequenceid=6, compaction requested=false 2023-07-15 18:13:08,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-15 18:13:08,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-15 18:13:08,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:08,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3513): Adding 83c5639bc741d79ce2b4c69e5abc7617 move to jenkins-hbase4.apache.org,43763,1689444781009 record at close sequenceid=6 2023-07-15 18:13:08,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,284 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=CLOSED 2023-07-15 18:13:08,284 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444788284"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444788284"}]},"ts":"1689444788284"} 2023-07-15 18:13:08,294 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-15 18:13:08,295 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; CloseRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,39349,1689444781225 in 280 msec 2023-07-15 18:13:08,296 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43763,1689444781009; forceNewPlan=false, retain=false 2023-07-15 18:13:08,447 INFO [jenkins-hbase4:42385] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 18:13:08,447 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:08,447 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444788447"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444788447"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444788447"}]},"ts":"1689444788447"} 2023-07-15 18:13:08,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,43763,1689444781009}] 2023-07-15 18:13:08,611 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,611 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 83c5639bc741d79ce2b4c69e5abc7617, NAME => 'hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:08,612 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,612 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:08,612 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,612 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,614 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,616 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:08,623 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:08,623 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 83c5639bc741d79ce2b4c69e5abc7617 columnFamilyName info 2023-07-15 18:13:08,643 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info/cdaf017d393d46f59089b5495bf62ca2 2023-07-15 18:13:08,643 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(310): Store=83c5639bc741d79ce2b4c69e5abc7617/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:08,644 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,647 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:08,653 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 83c5639bc741d79ce2b4c69e5abc7617; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10426418880, jitterRate=-0.028964072465896606}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:08,653 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:08,655 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617., pid=17, masterSystemTime=1689444788605 2023-07-15 18:13:08,657 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,657 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:08,658 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:08,658 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444788658"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444788658"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444788658"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444788658"}]},"ts":"1689444788658"} 2023-07-15 18:13:08,665 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-07-15 18:13:08,665 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,43763,1689444781009 in 209 msec 2023-07-15 18:13:08,667 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE in 667 msec 2023-07-15 18:13:08,852 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver 2023-07-15 18:13:08,854 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver Metrics about HBase MasterObservers 2023-07-15 18:13:08,854 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 18:13:08,854 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-15 18:13:08,855 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint 2023-07-15 18:13:08,855 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint Metrics about HBase MasterObservers 2023-07-15 18:13:09,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure.ProcedureSyncWait(216): waitFor pid=15 2023-07-15 18:13:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(261): All regions from [jenkins-hbase4.apache.org,39349,1689444781225] are moved back to default 2023-07-15 18:13:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(438): Move servers done: default => Group_testFallback_1624499712 2023-07-15 18:13:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveServers 2023-07-15 18:13:09,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:09,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:09,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=Group_testFallback_1624499712 2023-07-15 18:13:09,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:09,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'Group_testFallback', {NAME => 'f', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 18:13:09,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=Group_testFallback 2023-07-15 18:13:09,030 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=Group_testFallback execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 18:13:09,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "Group_testFallback" procId is: 18 2023-07-15 18:13:09,039 DEBUG [PEWorker-1] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:09,040 DEBUG [PEWorker-1] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:09,041 DEBUG [PEWorker-1] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:09,041 DEBUG [PEWorker-1] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:09,042 DEBUG [PEWorker-1] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:09,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-15 18:13:09,055 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=Group_testFallback execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 18:13:09,058 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,059 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d empty. 2023-07-15 18:13:09,059 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,060 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived Group_testFallback regions 2023-07-15 18:13:09,093 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/.tabledesc/.tableinfo.0000000001 2023-07-15 18:13:09,097 INFO [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(7675): creating {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='Group_testFallback', {NAME => 'f', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp 2023-07-15 18:13:09,131 DEBUG [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:09,131 DEBUG [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1604): Closing 516b5326c0ebf25df25947c8572ab42d, disabling compactions & flushes 2023-07-15 18:13:09,131 INFO [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1626): Closing region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,131 DEBUG [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,131 DEBUG [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1714): Acquired close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. after waiting 0 ms 2023-07-15 18:13:09,131 DEBUG [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1724): Updates disabled for region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,131 INFO [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1838): Closed Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,131 DEBUG [RegionOpenAndInit-Group_testFallback-pool-0] regionserver.HRegion(1558): Region close journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:09,135 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=Group_testFallback execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 18:13:09,137 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444789137"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444789137"}]},"ts":"1689444789137"} 2023-07-15 18:13:09,141 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 18:13:09,143 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=Group_testFallback execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 18:13:09,143 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"Group_testFallback","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444789143"}]},"ts":"1689444789143"} 2023-07-15 18:13:09,146 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=Group_testFallback, state=ENABLING in hbase:meta 2023-07-15 18:13:09,150 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 18:13:09,150 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 18:13:09,150 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 18:13:09,150 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 18:13:09,150 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN}] 2023-07-15 18:13:09,153 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN 2023-07-15 18:13:09,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-15 18:13:09,158 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43763,1689444781009; forceNewPlan=false, retain=false 2023-07-15 18:13:09,308 INFO [jenkins-hbase4:42385] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 18:13:09,310 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:09,310 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444789310"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444789310"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444789310"}]},"ts":"1689444789310"} 2023-07-15 18:13:09,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43763,1689444781009}] 2023-07-15 18:13:09,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-15 18:13:09,472 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:09,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table Group_testFallback 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:09,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,475 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,477 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:09,477 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:09,478 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 516b5326c0ebf25df25947c8572ab42d columnFamilyName f 2023-07-15 18:13:09,478 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(310): Store=516b5326c0ebf25df25947c8572ab42d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:09,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 18:13:09,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 516b5326c0ebf25df25947c8572ab42d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10490317600, jitterRate=-0.02301304042339325}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:09,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:09,492 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d., pid=20, masterSystemTime=1689444789465 2023-07-15 18:13:09,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,495 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:09,495 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444789495"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444789495"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444789495"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444789495"}]},"ts":"1689444789495"} 2023-07-15 18:13:09,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-15 18:13:09,502 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43763,1689444781009 in 185 msec 2023-07-15 18:13:09,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-15 18:13:09,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN in 352 msec 2023-07-15 18:13:09,507 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=Group_testFallback execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 18:13:09,507 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"Group_testFallback","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444789507"}]},"ts":"1689444789507"} 2023-07-15 18:13:09,509 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=Group_testFallback, state=ENABLED in hbase:meta 2023-07-15 18:13:09,513 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=Group_testFallback execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 18:13:09,515 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; CreateTableProcedure table=Group_testFallback in 488 msec 2023-07-15 18:13:09,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-15 18:13:09,660 INFO [Listener at localhost/46109] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:Group_testFallback, procId: 18 completed 2023-07-15 18:13:09,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(231): Client=jenkins//172.31.14.131 move tables [Group_testFallback] to rsgroup Group_testFallback_1624499712 2023-07-15 18:13:09,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:09,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:09,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:09,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:09,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(339): Moving region(s) for table Group_testFallback to RSGroup Group_testFallback_1624499712 2023-07-15 18:13:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(345): Moving region 516b5326c0ebf25df25947c8572ab42d to RSGroup Group_testFallback_1624499712 2023-07-15 18:13:09,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE 2023-07-15 18:13:09,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(286): Moving 1 region(s) to group Group_testFallback_1624499712, current retry=0 2023-07-15 18:13:09,684 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE 2023-07-15 18:13:09,685 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:09,685 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444789685"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444789685"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444789685"}]},"ts":"1689444789685"} 2023-07-15 18:13:09,688 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43763,1689444781009}] 2023-07-15 18:13:09,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 516b5326c0ebf25df25947c8572ab42d, disabling compactions & flushes 2023-07-15 18:13:09,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. after waiting 0 ms 2023-07-15 18:13:09,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-15 18:13:09,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:09,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:09,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3513): Adding 516b5326c0ebf25df25947c8572ab42d move to jenkins-hbase4.apache.org,39349,1689444781225 record at close sequenceid=2 2023-07-15 18:13:09,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:09,859 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSED 2023-07-15 18:13:09,859 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444789859"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444789859"}]},"ts":"1689444789859"} 2023-07-15 18:13:09,868 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-15 18:13:09,868 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43763,1689444781009 in 174 msec 2023-07-15 18:13:09,870 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39349,1689444781225; forceNewPlan=false, retain=false 2023-07-15 18:13:10,020 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,021 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444790020"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444790020"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444790020"}]},"ts":"1689444790020"} 2023-07-15 18:13:10,028 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,39349,1689444781225}] 2023-07-15 18:13:10,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:10,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:10,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table Group_testFallback 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:10,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,195 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,198 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:10,199 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:10,199 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 516b5326c0ebf25df25947c8572ab42d columnFamilyName f 2023-07-15 18:13:10,201 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(310): Store=516b5326c0ebf25df25947c8572ab42d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:10,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:10,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 516b5326c0ebf25df25947c8572ab42d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=11762335840, jitterRate=0.09545288980007172}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:10,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:10,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d., pid=23, masterSystemTime=1689444790185 2023-07-15 18:13:10,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:10,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:10,219 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,219 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444790219"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444790219"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444790219"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444790219"}]},"ts":"1689444790219"} 2023-07-15 18:13:10,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-07-15 18:13:10,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,39349,1689444781225 in 193 msec 2023-07-15 18:13:10,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE in 543 msec 2023-07-15 18:13:10,554 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-15 18:13:10,635 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'Group_testFallback' 2023-07-15 18:13:10,636 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-15 18:13:10,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure.ProcedureSyncWait(216): waitFor pid=21 2023-07-15 18:13:10,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(369): All regions from table(s) [Group_testFallback] moved to target group Group_testFallback_1624499712. 2023-07-15 18:13:10,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveTables 2023-07-15 18:13:10,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:10,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:10,693 DEBUG [Listener at localhost/46109] hbase.HBaseTestingUtility(3430): Waiting until all regions of table Group_testFallback get assigned. Timeout = 60000ms 2023-07-15 18:13:10,694 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:10,698 INFO [Listener at localhost/46109] hbase.HBaseTestingUtility(3484): All regions for table Group_testFallback assigned to meta. Checking AM states. 2023-07-15 18:13:10,699 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:10,699 INFO [Listener at localhost/46109] hbase.HBaseTestingUtility(3504): All regions for table Group_testFallback assigned. 2023-07-15 18:13:10,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=Group_testFallback_1624499712 2023-07-15 18:13:10,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:10,703 INFO [Listener at localhost/46109] assignment.AssignmentTestingUtil(82): KILL REGION SERVER jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,703 INFO [Listener at localhost/46109] hbase.MiniHBaseCluster(271): Killing jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,707 ERROR [Listener at localhost/46109] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,39349,1689444781225: Simulated kill ***** 2023-07-15 18:13:10,707 ERROR [Listener at localhost/46109] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver, org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 18:13:10,711 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 18:13:10,712 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 18:13:10,715 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 18:13:10,717 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 18:13:10,724 INFO [Listener at localhost/46109] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 985661440, "init": 513802240, "max": 2051014656, "used": 390713096 }, "NonHeapMemoryUsage": { "committed": 130506752, "init": 2555904, "max": -1, "used": 127839720 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 4, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 21668, "exceptions.RegionMovedException": 2, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 75, "ProcessCallTime_min": 0, "ProcessCallTime_max": 24, "ProcessCallTime_mean": 3, "ProcessCallTime_25th_percentile": 6, "ProcessCallTime_median": 12, "ProcessCallTime_75th_percentile": 18, "ProcessCallTime_90th_percentile": 21, "ProcessCallTime_95th_percentile": 22, "ProcessCallTime_98th_percentile": 23, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 75, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 75, "QueueCallTime_min": 0, "QueueCallTime_max": 3, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 1, "QueueCallTime_75th_percentile": 2, "QueueCallTime_90th_percentile": 2, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 2, "QueueCallTime_99th_percentile": 2, "QueueCallTime_99.9th_percentile": 2, "QueueCallTime_TimeRangeCount_0-1": 75, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 75, "TotalCallTime_min": 0, "TotalCallTime_max": 24, "TotalCallTime_mean": 3, "TotalCallTime_25th_percentile": 6, "TotalCallTime_median": 12, "TotalCallTime_75th_percentile": 18, "TotalCallTime_90th_percentile": 21, "TotalCallTime_95th_percentile": 22, "TotalCallTime_98th_percentile": 23, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 75, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 75, "ResponseSize_min": 0, "ResponseSize_max": 1611, "ResponseSize_mean": 103, "ResponseSize_25th_percentile": 402, "ResponseSize_median": 805, "ResponseSize_75th_percentile": 1208, "ResponseSize_90th_percentile": 1449, "ResponseSize_95th_percentile": 1530, "ResponseSize_98th_percentile": 1578, "ResponseSize_99th_percentile": 1594, "ResponseSize_99.9th_percentile": 1609, "ResponseSize_SizeRangeCount_0-10": 75, "exceptions.UnknownScannerException": 0, "exceptions": 3, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 75, "RequestSize_min": 80, "RequestSize_max": 630, "RequestSize_mean": 210, "RequestSize_25th_percentile": 217, "RequestSize_median": 355, "RequestSize_75th_percentile": 492, "RequestSize_90th_percentile": 575, "RequestSize_95th_percentile": 602, "RequestSize_98th_percentile": 619, "RequestSize_99th_percentile": 624, "RequestSize_99.9th_percentile": 629, "RequestSize_SizeRangeCount_0-10": 75, "sentBytes": 11656 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:62148", "tag.serverName": "jenkins-hbase4.apache.org,39349,1689444781225", "tag.clusterId": "1d505113-7488-4c3d-b421-4532e044e8dc", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 1, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689444781225, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2812, "localBytesRead": 2812, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 5415, "ByteBuffAllocatorPoolAllocationBytes": 399360, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 8, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 4, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 2, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 2, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 3, "Put_max": 13, "Put_mean": 8, "Put_25th_percentile": 5, "Put_median": 8, "Put_75th_percentile": 10, "Put_90th_percentile": 12, "Put_95th_percentile": 12, "Put_98th_percentile": 12, "Put_99th_percentile": 12, "Put_99.9th_percentile": 12, "Put_TimeRangeCount_0-1": 2, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 78, "FlushMemstoreSize_max": 78, "FlushMemstoreSize_mean": 78, "FlushMemstoreSize_25th_percentile": 78, "FlushMemstoreSize_median": 78, "FlushMemstoreSize_75th_percentile": 78, "FlushMemstoreSize_90th_percentile": 78, "FlushMemstoreSize_95th_percentile": 78, "FlushMemstoreSize_98th_percentile": 78, "FlushMemstoreSize_99th_percentile": 78, "FlushMemstoreSize_99.9th_percentile": 78, "FlushMemstoreSize_SizeRangeCount_0-10": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 4, "Get_min": 0, "Get_max": 2, "Get_mean": 1, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 1, "Get_90th_percentile": 1, "Get_95th_percentile": 1, "Get_98th_percentile": 1, "Get_99th_percentile": 1, "Get_99.9th_percentile": 1, "Get_TimeRangeCount_0-1": 4, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 70, "FlushTime_max": 70, "FlushTime_mean": 70, "FlushTime_25th_percentile": 70, "FlushTime_median": 70, "FlushTime_75th_percentile": 70, "FlushTime_90th_percentile": 70, "FlushTime_95th_percentile": 70, "FlushTime_98th_percentile": 70, "FlushTime_99th_percentile": 70, "FlushTime_99.9th_percentile": 70, "FlushTime_TimeRangeCount_0-1": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 2, "ScanSize_min": 86, "ScanSize_max": 86, "ScanSize_mean": 86, "ScanSize_25th_percentile": 86, "ScanSize_median": 86, "ScanSize_75th_percentile": 86, "ScanSize_90th_percentile": 86, "ScanSize_95th_percentile": 86, "ScanSize_98th_percentile": 86, "ScanSize_99th_percentile": 86, "ScanSize_99.9th_percentile": 86, "ScanSize_SizeRangeCount_0-10": 2, "slowGetCount": 0, "flushedMemstoreBytes": 78, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 4926, "FlushOutputSize_max": 4926, "FlushOutputSize_mean": 4926, "FlushOutputSize_25th_percentile": 4926, "FlushOutputSize_median": 4926, "FlushOutputSize_75th_percentile": 4926, "FlushOutputSize_90th_percentile": 4926, "FlushOutputSize_95th_percentile": 4926, "FlushOutputSize_98th_percentile": 4926, "FlushOutputSize_99th_percentile": 4926, "FlushOutputSize_99.9th_percentile": 4926, "FlushOutputSize_SizeRangeCount_0-10": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 43, "ServerReadQueryPerSecond_mean_rate": 7.0003451010592075, "ServerReadQueryPerSecond_1min_rate": 8.4, "ServerReadQueryPerSecond_5min_rate": 8.4, "ServerReadQueryPerSecond_15min_rate": 8.4, "ServerWriteQueryPerSecond_count": 55, "ServerWriteQueryPerSecond_mean_rate": 8.953934590798662, "ServerWriteQueryPerSecond_1min_rate": 9.0, "ServerWriteQueryPerSecond_5min_rate": 9.0, "ServerWriteQueryPerSecond_15min_rate": 9.0 } ] } 2023-07-15 18:13:10,730 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42385] master.MasterRpcServices(612): jenkins-hbase4.apache.org,39349,1689444781225 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,39349,1689444781225: Simulated kill ***** 2023-07-15 18:13:10,733 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39349,1689444781225' ***** 2023-07-15 18:13:10,734 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Simulated kill 2023-07-15 18:13:10,734 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:10,737 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:10,748 INFO [RS:1;jenkins-hbase4:39349] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@7a6bf43c{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:10,752 INFO [RS:1;jenkins-hbase4:39349] server.AbstractConnector(383): Stopped ServerConnector@56035e5{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:10,752 INFO [RS:1;jenkins-hbase4:39349] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:10,753 INFO [RS:1;jenkins-hbase4:39349] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@64c7beca{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:10,754 INFO [RS:1;jenkins-hbase4:39349] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@12839ff7{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:10,756 INFO [RS:1;jenkins-hbase4:39349] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:10,756 INFO [RS:1;jenkins-hbase4:39349] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 18:13:10,756 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:10,757 INFO [RS:1;jenkins-hbase4:39349] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 18:13:10,757 DEBUG [RS:1;jenkins-hbase4:39349] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e1b858b to 127.0.0.1:62148 2023-07-15 18:13:10,757 DEBUG [RS:1;jenkins-hbase4:39349] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:10,770 DEBUG [RS:1;jenkins-hbase4:39349] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:10,771 INFO [RS:1;jenkins-hbase4:39349] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:10,772 INFO [RS:1;jenkins-hbase4:39349] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39349 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:10,781 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 2023-07-15 18:13:10,782 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:10,782 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:10,785 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39349,1689444781225] 2023-07-15 18:13:10,785 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39349,1689444781225; numProcessing=1 2023-07-15 18:13:10,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,787 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39349,1689444781225 already deleted, retry=false 2023-07-15 18:13:10,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:10,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:10,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:10,787 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39349,1689444781225 on jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:10,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:10,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:10,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:10,792 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 znode expired, triggering replicatorRemoved event 2023-07-15 18:13:10,792 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 znode expired, triggering replicatorRemoved event 2023-07-15 18:13:10,794 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39349,1689444781225 znode expired, triggering replicatorRemoved event 2023-07-15 18:13:10,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:10,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:10,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:10,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:10,803 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39349,1689444781225, splitWal=true, meta=false 2023-07-15 18:13:10,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,803 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=24 for jenkins-hbase4.apache.org,39349,1689444781225 (carryingMeta=false) jenkins-hbase4.apache.org,39349,1689444781225/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1b6d7d5e[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 18:13:10,804 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:10,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:10,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:10,810 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=24, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39349,1689444781225, splitWal=true, meta=false 2023-07-15 18:13:10,811 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:10,811 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:10,812 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39349,1689444781225 had 1 regions 2023-07-15 18:13:10,812 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:10,812 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:10,813 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:10,814 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=24, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39349,1689444781225, splitWal=true, meta=false, isMeta: false 2023-07-15 18:13:10,816 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 2 2023-07-15 18:13:10,818 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting 2023-07-15 18:13:10,820 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:10,821 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,39349,1689444781225 WAL count=1, meta=false 2023-07-15 18:13:10,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684}] 2023-07-15 18:13:10,839 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:10,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, worker=jenkins-hbase4.apache.org,42063,1689444786274}] 2023-07-15 18:13:10,841 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:10,959 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:10,960 INFO [RS:1;jenkins-hbase4:39349] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39349,1689444781225; zookeeper connection closed. 2023-07-15 18:13:10,960 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:39349-0x1016a3001d00002, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:10,961 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@720c3efd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@720c3efd 2023-07-15 18:13:11,010 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42063] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=26 2023-07-15 18:13:11,043 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, size=1.7 K (1746bytes) 2023-07-15 18:13:11,044 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 2023-07-15 18:13:11,050 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 after 6ms 2023-07-15 18:13:11,056 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 took 16ms 2023-07-15 18:13:11,078 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 83c5639bc741d79ce2b4c69e5abc7617: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-15 18:13:11,084 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 516b5326c0ebf25df25947c8572ab42d: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 18446744073709551615 } 2023-07-15 18:13:11,084 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 so closing down 2023-07-15 18:13:11,084 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 18:13:11,096 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-15 18:13:11,099 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684.temp 2023-07-15 18:13:11,100 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 18:13:11,200 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 18:13:11,203 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684.temp to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000005 2023-07-15 18:13:11,203 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 7 edits across 1 Regions in 145 ms; skipped=5; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, size=1.7 K, length=1746, corrupted=false, cancelled=false 2023-07-15 18:13:11,203 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, journal: Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, size=1.7 K (1746bytes) at 1689444791043Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 so closing down at 1689444791084 (+41 ms)Creating recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684.temp at 1689444791099 (+15 ms)3 split writer threads finished at 1689444791100 (+1 ms)Closed recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689444791201 (+101 ms)Rename recovered edits hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684.temp to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000005 at 1689444791203 (+2 ms)Processed 7 edits across 1 Regions in 145 ms; skipped=5; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, size=1.7 K, length=1746, corrupted=false, cancelled=false at 1689444791203 2023-07-15 18:13:11,204 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 2023-07-15 18:13:11,208 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=26 2023-07-15 18:13:11,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster(3641): Remote procedure done, pid=26 2023-07-15 18:13:11,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] wal.WALSplitUtil(141): Moved hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting/jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684 to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:11,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-15 18:13:11,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, worker=jenkins-hbase4.apache.org,42063,1689444786274 in 390 msec 2023-07-15 18:13:11,236 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,242 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-15 18:13:11,242 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39349%2C1689444781225.1689444784684, worker=jenkins-hbase4.apache.org,42063,1689444786274 in 414 msec 2023-07-15 18:13:11,245 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting dir is empty, no logs to split. 2023-07-15 18:13:11,245 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,39349,1689444781225 WAL count=0, meta=false 2023-07-15 18:13:11,245 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39349,1689444781225 WAL splitting is done? wals=0, meta=false 2023-07-15 18:13:11,263 INFO [PEWorker-1] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,39349,1689444781225 failed, ignore...File hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,39349,1689444781225-splitting does not exist. 2023-07-15 18:13:11,266 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN}] 2023-07-15 18:13:11,269 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN 2023-07-15 18:13:11,271 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=27, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:11,421 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 18:13:11,421 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 18:13:11,421 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 18:13:11,421 DEBUG [jenkins-hbase4:42385] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 18:13:11,422 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,422 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444791422"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444791422"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444791422"}]},"ts":"1689444791422"} 2023-07-15 18:13:11,424 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,42063,1689444786274}] 2023-07-15 18:13:11,513 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 18:13:11,513 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 18:13:11,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:11,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:11,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table Group_testFallback 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:11,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,584 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,585 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:11,585 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:11,586 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 516b5326c0ebf25df25947c8572ab42d columnFamilyName f 2023-07-15 18:13:11,587 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(310): Store=516b5326c0ebf25df25947c8572ab42d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:11,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:11,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 516b5326c0ebf25df25947c8572ab42d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=11065985120, jitterRate=0.030600175261497498}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:11,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:11,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d., pid=28, masterSystemTime=1689444791577 2023-07-15 18:13:11,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:11,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:11,599 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,599 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444791598"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444791598"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444791598"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444791598"}]},"ts":"1689444791598"} 2023-07-15 18:13:11,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-15 18:13:11,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,42063,1689444786274 in 177 msec 2023-07-15 18:13:11,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-15 18:13:11,607 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39349,1689444781225 after splitting done 2023-07-15 18:13:11,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN in 337 msec 2023-07-15 18:13:11,607 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39349,1689444781225 from processing; numProcessing=0 2023-07-15 18:13:11,609 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39349,1689444781225, splitWal=true, meta=false in 814 msec 2023-07-15 18:13:11,643 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:11,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:11,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:11,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:11,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:11,648 INFO [Listener at localhost/46109] assignment.AssignmentTestingUtil(82): KILL REGION SERVER jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,648 INFO [Listener at localhost/46109] hbase.MiniHBaseCluster(271): Killing jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,649 ERROR [Listener at localhost/46109] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,42063,1689444786274: Simulated kill ***** 2023-07-15 18:13:11,649 ERROR [Listener at localhost/46109] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver, org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 18:13:11,649 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 18:13:11,649 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 18:13:11,650 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 18:13:11,651 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 18:13:11,653 INFO [Listener at localhost/46109] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 985661440, "init": 513802240, "max": 2051014656, "used": 425615672 }, "NonHeapMemoryUsage": { "committed": 133128192, "init": 2555904, "max": -1, "used": 130626544 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 4, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 21668, "exceptions.RegionMovedException": 2, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 75, "ProcessCallTime_min": 0, "ProcessCallTime_max": 24, "ProcessCallTime_mean": 3, "ProcessCallTime_25th_percentile": 6, "ProcessCallTime_median": 12, "ProcessCallTime_75th_percentile": 18, "ProcessCallTime_90th_percentile": 21, "ProcessCallTime_95th_percentile": 22, "ProcessCallTime_98th_percentile": 23, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 75, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 75, "QueueCallTime_min": 0, "QueueCallTime_max": 3, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 1, "QueueCallTime_75th_percentile": 2, "QueueCallTime_90th_percentile": 2, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 2, "QueueCallTime_99th_percentile": 2, "QueueCallTime_99.9th_percentile": 2, "QueueCallTime_TimeRangeCount_0-1": 75, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 75, "TotalCallTime_min": 0, "TotalCallTime_max": 24, "TotalCallTime_mean": 3, "TotalCallTime_25th_percentile": 6, "TotalCallTime_median": 12, "TotalCallTime_75th_percentile": 18, "TotalCallTime_90th_percentile": 21, "TotalCallTime_95th_percentile": 22, "TotalCallTime_98th_percentile": 23, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 75, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 75, "ResponseSize_min": 0, "ResponseSize_max": 1611, "ResponseSize_mean": 103, "ResponseSize_25th_percentile": 402, "ResponseSize_median": 805, "ResponseSize_75th_percentile": 1208, "ResponseSize_90th_percentile": 1449, "ResponseSize_95th_percentile": 1530, "ResponseSize_98th_percentile": 1578, "ResponseSize_99th_percentile": 1594, "ResponseSize_99.9th_percentile": 1609, "ResponseSize_SizeRangeCount_0-10": 75, "exceptions.UnknownScannerException": 0, "exceptions": 3, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 75, "RequestSize_min": 80, "RequestSize_max": 630, "RequestSize_mean": 210, "RequestSize_25th_percentile": 217, "RequestSize_median": 355, "RequestSize_75th_percentile": 492, "RequestSize_90th_percentile": 575, "RequestSize_95th_percentile": 602, "RequestSize_98th_percentile": 619, "RequestSize_99th_percentile": 624, "RequestSize_99.9th_percentile": 629, "RequestSize_SizeRangeCount_0-10": 75, "sentBytes": 11656 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:62148", "tag.serverName": "jenkins-hbase4.apache.org,39349,1689444781225", "tag.clusterId": "1d505113-7488-4c3d-b421-4532e044e8dc", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 1, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689444781225, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2812, "localBytesRead": 2812, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 5415, "ByteBuffAllocatorPoolAllocationBytes": 399360, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 8, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 4, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 2, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 2, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 3, "Put_max": 13, "Put_mean": 8, "Put_25th_percentile": 5, "Put_median": 8, "Put_75th_percentile": 10, "Put_90th_percentile": 12, "Put_95th_percentile": 12, "Put_98th_percentile": 12, "Put_99th_percentile": 12, "Put_99.9th_percentile": 12, "Put_TimeRangeCount_0-1": 2, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 78, "FlushMemstoreSize_max": 78, "FlushMemstoreSize_mean": 78, "FlushMemstoreSize_25th_percentile": 78, "FlushMemstoreSize_median": 78, "FlushMemstoreSize_75th_percentile": 78, "FlushMemstoreSize_90th_percentile": 78, "FlushMemstoreSize_95th_percentile": 78, "FlushMemstoreSize_98th_percentile": 78, "FlushMemstoreSize_99th_percentile": 78, "FlushMemstoreSize_99.9th_percentile": 78, "FlushMemstoreSize_SizeRangeCount_0-10": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 4, "Get_min": 0, "Get_max": 2, "Get_mean": 1, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 1, "Get_90th_percentile": 1, "Get_95th_percentile": 1, "Get_98th_percentile": 1, "Get_99th_percentile": 1, "Get_99.9th_percentile": 1, "Get_TimeRangeCount_0-1": 4, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 70, "FlushTime_max": 70, "FlushTime_mean": 70, "FlushTime_25th_percentile": 70, "FlushTime_median": 70, "FlushTime_75th_percentile": 70, "FlushTime_90th_percentile": 70, "FlushTime_95th_percentile": 70, "FlushTime_98th_percentile": 70, "FlushTime_99th_percentile": 70, "FlushTime_99.9th_percentile": 70, "FlushTime_TimeRangeCount_0-1": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 2, "ScanSize_min": 86, "ScanSize_max": 86, "ScanSize_mean": 86, "ScanSize_25th_percentile": 86, "ScanSize_median": 86, "ScanSize_75th_percentile": 86, "ScanSize_90th_percentile": 86, "ScanSize_95th_percentile": 86, "ScanSize_98th_percentile": 86, "ScanSize_99th_percentile": 86, "ScanSize_99.9th_percentile": 86, "ScanSize_SizeRangeCount_0-10": 2, "slowGetCount": 0, "flushedMemstoreBytes": 78, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 4926, "FlushOutputSize_max": 4926, "FlushOutputSize_mean": 4926, "FlushOutputSize_25th_percentile": 4926, "FlushOutputSize_median": 4926, "FlushOutputSize_75th_percentile": 4926, "FlushOutputSize_90th_percentile": 4926, "FlushOutputSize_95th_percentile": 4926, "FlushOutputSize_98th_percentile": 4926, "FlushOutputSize_99th_percentile": 4926, "FlushOutputSize_99.9th_percentile": 4926, "FlushOutputSize_SizeRangeCount_0-10": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 43, "ServerReadQueryPerSecond_mean_rate": 7.0003451010592075, "ServerReadQueryPerSecond_1min_rate": 8.4, "ServerReadQueryPerSecond_5min_rate": 8.4, "ServerReadQueryPerSecond_15min_rate": 8.4, "ServerWriteQueryPerSecond_count": 55, "ServerWriteQueryPerSecond_mean_rate": 8.953934590798662, "ServerWriteQueryPerSecond_1min_rate": 9.0, "ServerWriteQueryPerSecond_5min_rate": 9.0, "ServerWriteQueryPerSecond_15min_rate": 9.0 } ] } 2023-07-15 18:13:11,654 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42385] master.MasterRpcServices(612): jenkins-hbase4.apache.org,42063,1689444786274 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,42063,1689444786274: Simulated kill ***** 2023-07-15 18:13:11,655 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42063,1689444786274' ***** 2023-07-15 18:13:11,656 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Simulated kill 2023-07-15 18:13:11,656 INFO [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:11,656 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:11,661 INFO [RS:3;jenkins-hbase4:42063] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@407fb65b{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:11,661 INFO [RS:3;jenkins-hbase4:42063] server.AbstractConnector(383): Stopped ServerConnector@26f2f4a9{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:11,661 INFO [RS:3;jenkins-hbase4:42063] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:11,662 INFO [RS:3;jenkins-hbase4:42063] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@745540d2{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:11,663 INFO [RS:3;jenkins-hbase4:42063] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@4deaba7a{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:11,663 INFO [RS:3;jenkins-hbase4:42063] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:11,663 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:11,663 INFO [RS:3;jenkins-hbase4:42063] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 18:13:11,664 INFO [RS:3;jenkins-hbase4:42063] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 18:13:11,664 DEBUG [RS:3;jenkins-hbase4:42063] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5abaedce to 127.0.0.1:62148 2023-07-15 18:13:11,664 DEBUG [RS:3;jenkins-hbase4:42063] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:11,669 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 not finished, retry = 0 2023-07-15 18:13:11,716 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:11,773 DEBUG [RS:3;jenkins-hbase4:42063] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:11,773 INFO [RS:3;jenkins-hbase4:42063] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:11,773 INFO [RS:3;jenkins-hbase4:42063] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42063 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,783 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,785 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42063,1689444786274] 2023-07-15 18:13:11,785 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42063,1689444786274; numProcessing=1 2023-07-15 18:13:11,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,786 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42063,1689444786274 already deleted, retry=false 2023-07-15 18:13:11,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,787 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 znode expired, triggering replicatorRemoved event 2023-07-15 18:13:11,787 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,42063,1689444786274 znode expired, triggering replicatorRemoved event 2023-07-15 18:13:11,786 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,42063,1689444786274 on jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:11,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,788 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=29, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,42063,1689444786274, splitWal=true, meta=false 2023-07-15 18:13:11,789 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=29 for jenkins-hbase4.apache.org,42063,1689444786274 (carryingMeta=false) jenkins-hbase4.apache.org,42063,1689444786274/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@24bc74a4[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 18:13:11,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,789 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:11,790 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=29, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42063,1689444786274, splitWal=true, meta=false 2023-07-15 18:13:11,790 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42063 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42063 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:11,792 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42063 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42063 2023-07-15 18:13:11,792 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,42063,1689444786274 had 2 regions 2023-07-15 18:13:11,794 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=29, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42063,1689444786274, splitWal=true, meta=false, isMeta: false 2023-07-15 18:13:11,796 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting 2023-07-15 18:13:11,797 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,42063,1689444786274 WAL count=1, meta=false 2023-07-15 18:13:11,797 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652}] 2023-07-15 18:13:11,799 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,800 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=30, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, worker=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:11,856 INFO [Listener at localhost/46109] assignment.AssignmentTestingUtil(82): KILL REGION SERVER jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,857 INFO [Listener at localhost/46109] hbase.MiniHBaseCluster(271): Killing jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,857 ERROR [Listener at localhost/46109] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,43763,1689444781009: Simulated kill ***** 2023-07-15 18:13:11,857 ERROR [Listener at localhost/46109] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver, org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 18:13:11,857 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 18:13:11,857 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 18:13:11,858 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 18:13:11,858 DEBUG [Listener at localhost/46109] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 18:13:11,859 INFO [Listener at localhost/46109] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 985661440, "init": 513802240, "max": 2051014656, "used": 427074944 }, "NonHeapMemoryUsage": { "committed": 133652480, "init": 2555904, "max": -1, "used": 130890776 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 4, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 21668, "exceptions.RegionMovedException": 2, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 75, "ProcessCallTime_min": 0, "ProcessCallTime_max": 24, "ProcessCallTime_mean": 3, "ProcessCallTime_25th_percentile": 6, "ProcessCallTime_median": 12, "ProcessCallTime_75th_percentile": 18, "ProcessCallTime_90th_percentile": 21, "ProcessCallTime_95th_percentile": 22, "ProcessCallTime_98th_percentile": 23, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 75, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 75, "QueueCallTime_min": 0, "QueueCallTime_max": 3, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 1, "QueueCallTime_75th_percentile": 2, "QueueCallTime_90th_percentile": 2, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 2, "QueueCallTime_99th_percentile": 2, "QueueCallTime_99.9th_percentile": 2, "QueueCallTime_TimeRangeCount_0-1": 75, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 75, "TotalCallTime_min": 0, "TotalCallTime_max": 24, "TotalCallTime_mean": 3, "TotalCallTime_25th_percentile": 6, "TotalCallTime_median": 12, "TotalCallTime_75th_percentile": 18, "TotalCallTime_90th_percentile": 21, "TotalCallTime_95th_percentile": 22, "TotalCallTime_98th_percentile": 23, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 75, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 75, "ResponseSize_min": 0, "ResponseSize_max": 1611, "ResponseSize_mean": 103, "ResponseSize_25th_percentile": 402, "ResponseSize_median": 805, "ResponseSize_75th_percentile": 1208, "ResponseSize_90th_percentile": 1449, "ResponseSize_95th_percentile": 1530, "ResponseSize_98th_percentile": 1578, "ResponseSize_99th_percentile": 1594, "ResponseSize_99.9th_percentile": 1609, "ResponseSize_SizeRangeCount_0-10": 75, "exceptions.UnknownScannerException": 0, "exceptions": 3, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 75, "RequestSize_min": 80, "RequestSize_max": 630, "RequestSize_mean": 210, "RequestSize_25th_percentile": 217, "RequestSize_median": 355, "RequestSize_75th_percentile": 492, "RequestSize_90th_percentile": 575, "RequestSize_95th_percentile": 602, "RequestSize_98th_percentile": 619, "RequestSize_99th_percentile": 624, "RequestSize_99.9th_percentile": 629, "RequestSize_SizeRangeCount_0-10": 75, "sentBytes": 11656 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:62148", "tag.serverName": "jenkins-hbase4.apache.org,39349,1689444781225", "tag.clusterId": "1d505113-7488-4c3d-b421-4532e044e8dc", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 1, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689444781225, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2812, "localBytesRead": 2812, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 5415, "ByteBuffAllocatorPoolAllocationBytes": 399360, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 8, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 4, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 2, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 2, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 3, "Put_max": 13, "Put_mean": 8, "Put_25th_percentile": 5, "Put_median": 8, "Put_75th_percentile": 10, "Put_90th_percentile": 12, "Put_95th_percentile": 12, "Put_98th_percentile": 12, "Put_99th_percentile": 12, "Put_99.9th_percentile": 12, "Put_TimeRangeCount_0-1": 2, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 78, "FlushMemstoreSize_max": 78, "FlushMemstoreSize_mean": 78, "FlushMemstoreSize_25th_percentile": 78, "FlushMemstoreSize_median": 78, "FlushMemstoreSize_75th_percentile": 78, "FlushMemstoreSize_90th_percentile": 78, "FlushMemstoreSize_95th_percentile": 78, "FlushMemstoreSize_98th_percentile": 78, "FlushMemstoreSize_99th_percentile": 78, "FlushMemstoreSize_99.9th_percentile": 78, "FlushMemstoreSize_SizeRangeCount_0-10": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 4, "Get_min": 0, "Get_max": 2, "Get_mean": 1, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 1, "Get_90th_percentile": 1, "Get_95th_percentile": 1, "Get_98th_percentile": 1, "Get_99th_percentile": 1, "Get_99.9th_percentile": 1, "Get_TimeRangeCount_0-1": 4, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 70, "FlushTime_max": 70, "FlushTime_mean": 70, "FlushTime_25th_percentile": 70, "FlushTime_median": 70, "FlushTime_75th_percentile": 70, "FlushTime_90th_percentile": 70, "FlushTime_95th_percentile": 70, "FlushTime_98th_percentile": 70, "FlushTime_99th_percentile": 70, "FlushTime_99.9th_percentile": 70, "FlushTime_TimeRangeCount_0-1": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 2, "ScanSize_min": 86, "ScanSize_max": 86, "ScanSize_mean": 86, "ScanSize_25th_percentile": 86, "ScanSize_median": 86, "ScanSize_75th_percentile": 86, "ScanSize_90th_percentile": 86, "ScanSize_95th_percentile": 86, "ScanSize_98th_percentile": 86, "ScanSize_99th_percentile": 86, "ScanSize_99.9th_percentile": 86, "ScanSize_SizeRangeCount_0-10": 2, "slowGetCount": 0, "flushedMemstoreBytes": 78, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 4926, "FlushOutputSize_max": 4926, "FlushOutputSize_mean": 4926, "FlushOutputSize_25th_percentile": 4926, "FlushOutputSize_median": 4926, "FlushOutputSize_75th_percentile": 4926, "FlushOutputSize_90th_percentile": 4926, "FlushOutputSize_95th_percentile": 4926, "FlushOutputSize_98th_percentile": 4926, "FlushOutputSize_99th_percentile": 4926, "FlushOutputSize_99.9th_percentile": 4926, "FlushOutputSize_SizeRangeCount_0-10": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 43, "ServerReadQueryPerSecond_mean_rate": 7.0003451010592075, "ServerReadQueryPerSecond_1min_rate": 8.4, "ServerReadQueryPerSecond_5min_rate": 8.4, "ServerReadQueryPerSecond_15min_rate": 8.4, "ServerWriteQueryPerSecond_count": 55, "ServerWriteQueryPerSecond_mean_rate": 8.953934590798662, "ServerWriteQueryPerSecond_1min_rate": 9.0, "ServerWriteQueryPerSecond_5min_rate": 9.0, "ServerWriteQueryPerSecond_15min_rate": 9.0 } ] } 2023-07-15 18:13:11,860 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42385] master.MasterRpcServices(612): jenkins-hbase4.apache.org,43763,1689444781009 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,43763,1689444781009: Simulated kill ***** 2023-07-15 18:13:11,861 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43763,1689444781009' ***** 2023-07-15 18:13:11,862 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Simulated kill 2023-07-15 18:13:11,862 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:11,862 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:11,866 INFO [RS:0;jenkins-hbase4:43763] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@32d72d2f{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:11,866 INFO [RS:0;jenkins-hbase4:43763] server.AbstractConnector(383): Stopped ServerConnector@5b9eff03{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:11,866 INFO [RS:0;jenkins-hbase4:43763] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:11,867 INFO [RS:0;jenkins-hbase4:43763] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@4e584a37{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:11,867 INFO [RS:0;jenkins-hbase4:43763] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@639f3f91{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:11,868 INFO [RS:0;jenkins-hbase4:43763] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:11,868 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:11,868 INFO [RS:0;jenkins-hbase4:43763] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 18:13:11,868 INFO [RS:0;jenkins-hbase4:43763] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 18:13:11,868 DEBUG [RS:0;jenkins-hbase4:43763] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62c48004 to 127.0.0.1:62148 2023-07-15 18:13:11,868 DEBUG [RS:0;jenkins-hbase4:43763] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:11,877 DEBUG [RS:0;jenkins-hbase4:43763] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:11,877 INFO [RS:0;jenkins-hbase4:43763] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:11,878 INFO [RS:0;jenkins-hbase4:43763] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43763 2023-07-15 18:13:11,880 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,880 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,880 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,880 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 2023-07-15 18:13:11,880 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:11,881 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43763,1689444781009] 2023-07-15 18:13:11,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,882 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43763,1689444781009; numProcessing=2 2023-07-15 18:13:11,882 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,43763,1689444781009 znode expired, triggering replicatorRemoved event 2023-07-15 18:13:11,883 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43763,1689444781009 already deleted, retry=false 2023-07-15 18:13:11,883 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,43763,1689444781009 on jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:11,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,884 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=32, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,43763,1689444781009, splitWal=true, meta=true 2023-07-15 18:13:11,884 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=32 for jenkins-hbase4.apache.org,43763,1689444781009 (carryingMeta=true) jenkins-hbase4.apache.org,43763,1689444781009/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@54cb7ae0[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 18:13:11,885 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:11,885 INFO [RS:3;jenkins-hbase4:42063] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42063,1689444786274; zookeeper connection closed. 2023-07-15 18:13:11,885 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:42063-0x1016a3001d0000b, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:11,886 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6b58ac61] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6b58ac61 2023-07-15 18:13:11,886 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=32, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43763,1689444781009, splitWal=true, meta=true 2023-07-15 18:13:11,887 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=32, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43763,1689444781009, splitWal=true, meta=true, isMeta: true 2023-07-15 18:13:11,889 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting 2023-07-15 18:13:11,890 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,43763,1689444781009 WAL count=1, meta=true 2023-07-15 18:13:11,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta}] 2023-07-15 18:13:11,892 WARN [RS-EventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43763 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43763 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:11,893 DEBUG [RS-EventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43763 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43763 2023-07-15 18:13:11,895 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:11,896 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=33, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, worker=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:11,921 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:11,953 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33055] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=34 2023-07-15 18:13:11,953 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33055] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=31 2023-07-15 18:13:11,962 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:11,966 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, size=3.0 K (3109bytes) 2023-07-15 18:13:11,966 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 2023-07-15 18:13:11,966 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 after 0ms 2023-07-15 18:13:11,967 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, size=11.1 K (11383bytes) 2023-07-15 18:13:11,967 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta 2023-07-15 18:13:11,968 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta after 1ms 2023-07-15 18:13:11,969 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 took 4ms 2023-07-15 18:13:11,971 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta took 4ms 2023-07-15 18:13:11,973 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 994da785cf935856bc388ebe43136633: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "m" sequence_id: 14 } 2023-07-15 18:13:11,976 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 516b5326c0ebf25df25947c8572ab42d: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 18446744073709551615 } 2023-07-15 18:13:11,976 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 so closing down 2023-07-15 18:13:11,976 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 18:13:11,977 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-15 18:13:11,979 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000015-jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652.temp 2023-07-15 18:13:11,980 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 18:13:11,982 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta so closing down 2023-07-15 18:13:11,982 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 18:13:11,985 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta.temp 2023-07-15 18:13:11,985 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:11,985 INFO [RS:0;jenkins-hbase4:43763] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43763,1689444781009; zookeeper connection closed. 2023-07-15 18:13:11,985 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43763-0x1016a3001d00001, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:11,986 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 18:13:11,986 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5a8c764c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5a8c764c 2023-07-15 18:13:11,999 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000015-jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652.temp (wrote 5 edits, skipped 0 edits in 0 ms) 2023-07-15 18:13:11,999 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43763 this server is in the failed servers list 2023-07-15 18:13:12,002 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000015-jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652.temp to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000019 2023-07-15 18:13:12,003 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 7 edits across 1 Regions in 33 ms; skipped=2; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, size=3.0 K, length=3109, corrupted=false, cancelled=false 2023-07-15 18:13:12,003 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, journal: Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, size=3.0 K (3109bytes) at 1689444791966Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 so closing down at 1689444791976 (+10 ms)Creating recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000015-jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652.temp at 1689444791979 (+3 ms)3 split writer threads finished at 1689444791980 (+1 ms)Closed recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000015-jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652.temp (wrote 5 edits, skipped 0 edits in 0 ms) at 1689444791999 (+19 ms)Rename recovered edits hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000015-jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652.temp to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000019 at 1689444792002 (+3 ms)Processed 7 edits across 1 Regions in 33 ms; skipped=2; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, size=3.0 K, length=3109, corrupted=false, cancelled=false at 1689444792003 (+1 ms) 2023-07-15 18:13:12,003 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 2023-07-15 18:13:12,003 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=31 2023-07-15 18:13:12,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster(3641): Remote procedure done, pid=31 2023-07-15 18:13:12,005 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta.temp (wrote 29 edits, skipped 0 edits in 0 ms) 2023-07-15 18:13:12,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] wal.WALSplitUtil(141): Moved hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting/jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652 to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:12,007 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta.temp to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000032 2023-07-15 18:13:12,008 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 30 edits across 1 Regions in 34 ms; skipped=1; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, size=11.1 K, length=11383, corrupted=false, cancelled=false 2023-07-15 18:13:12,008 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, journal: Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, size=11.1 K (11383bytes) at 1689444791967Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta so closing down at 1689444791982 (+15 ms)Creating recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta.temp at 1689444791985 (+3 ms)3 split writer threads finished at 1689444791986 (+1 ms)Closed recovered edits writer path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta.temp (wrote 29 edits, skipped 0 edits in 0 ms) at 1689444792005 (+19 ms)Rename recovered edits hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta.temp to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000032 at 1689444792007 (+2 ms)Processed 30 edits across 1 Regions in 34 ms; skipped=1; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, size=11.1 K, length=11383, corrupted=false, cancelled=false at 1689444792008 (+1 ms) 2023-07-15 18:13:12,008 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta 2023-07-15 18:13:12,008 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=34 2023-07-15 18:13:12,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster(3641): Remote procedure done, pid=34 2023-07-15 18:13:12,011 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=30 2023-07-15 18:13:12,011 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=30, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, worker=jenkins-hbase4.apache.org,33055,1689444781474 in 208 msec 2023-07-15 18:13:12,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] wal.WALSplitUtil(141): Moved hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:12,013 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-15 18:13:12,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=33 2023-07-15 18:13:12,016 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C42063%2C1689444786274.1689444786652, worker=jenkins-hbase4.apache.org,33055,1689444781474 in 216 msec 2023-07-15 18:13:12,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=33, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, worker=jenkins-hbase4.apache.org,33055,1689444781474 in 117 msec 2023-07-15 18:13:12,017 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,018 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting dir is empty, no logs to split. 2023-07-15 18:13:12,018 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,42063,1689444786274 WAL count=0, meta=false 2023-07-15 18:13:12,018 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42063,1689444786274 WAL splitting is done? wals=0, meta=false 2023-07-15 18:13:12,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-07-15 18:13:12,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.meta.1689444784934.meta, worker=jenkins-hbase4.apache.org,33055,1689444781474 in 127 msec 2023-07-15 18:13:12,021 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,42063,1689444786274 failed, ignore...File hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,42063,1689444786274-splitting does not exist. 2023-07-15 18:13:12,021 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting dir is empty, no logs to split. 2023-07-15 18:13:12,022 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,43763,1689444781009 WAL count=0, meta=true 2023-07-15 18:13:12,022 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43763,1689444781009 WAL splitting is done? wals=0, meta=true 2023-07-15 18:13:12,022 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=29, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN}, {pid=36, ppid=29, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN}] 2023-07-15 18:13:12,023 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=32, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 18:13:12,023 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=35, ppid=29, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN 2023-07-15 18:13:12,023 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=29, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN 2023-07-15 18:13:12,024 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=35, ppid=29, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:12,024 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=32, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 18:13:12,024 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=29, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:12,025 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=32, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:12,026 DEBUG [jenkins-hbase4:42385] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 18:13:12,026 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=35 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,026 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444792026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444792026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444792026"}]},"ts":"1689444792026"} 2023-07-15 18:13:12,026 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33055,1689444781474, state=OPENING 2023-07-15 18:13:12,026 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,026 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444792026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444792026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444792026"}]},"ts":"1689444792026"} 2023-07-15 18:13:12,028 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 18:13:12,028 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:12,028 WARN [ReadOnlyZKClient-127.0.0.1:62148@0x0689ffaf] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 18:13:12,028 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=37, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:12,030 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 293 connection: 172.31.14.131:33790 deadline: 1689444852029, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,030 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 275 connection: 172.31.14.131:33790 deadline: 1689444852029, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,132 WARN [ReadOnlyZKClient-127.0.0.1:62148@0x0689ffaf] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 18:13:12,132 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 293 connection: 172.31.14.131:33790 deadline: 1689444852132, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,133 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 275 connection: 172.31.14.131:33790 deadline: 1689444852132, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,186 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 18:13:12,186 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:12,188 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33055%2C1689444781474.meta, suffix=.meta, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,33055,1689444781474, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:12,202 WARN [ReadOnlyZKClient-127.0.0.1:62148@0x0689ffaf] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 18:13:12,202 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33055] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Get size: 88 connection: 172.31.14.131:33790 deadline: 1689444852202, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,205 DEBUG [RS-EventLoopGroup-7-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:12,206 DEBUG [RS-EventLoopGroup-7-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:12,206 DEBUG [RS-EventLoopGroup-7-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:12,210 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,33055,1689444781474/jenkins-hbase4.apache.org%2C33055%2C1689444781474.meta.1689444792189.meta 2023-07-15 18:13:12,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK]] 2023-07-15 18:13:12,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:12,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:12,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 18:13:12,212 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 18:13:12,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 18:13:12,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:12,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 18:13:12,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 18:13:12,214 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 18:13:12,215 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:12,215 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:12,216 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 18:13:12,217 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:12,217 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 18:13:12,218 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:12,218 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:12,220 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 18:13:12,221 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:12,221 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 18:13:12,223 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:12,223 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:12,223 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 18:13:12,224 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:12,225 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:12,228 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:12,229 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000032 2023-07-15 18:13:12,242 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 83, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=32, path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000032 2023-07-15 18:13:12,242 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=9.45 KB heapSize=16.34 KB 2023-07-15 18:13:12,264 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=9.17 KB at sequenceid=32 (bloomFilter=false), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/info/a11b44c324064f2c984d2b9f474249b3 2023-07-15 18:13:12,293 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=284 B at sequenceid=32 (bloomFilter=false), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/table/18ae753c949e461f952f90d512445679 2023-07-15 18:13:12,301 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/info/a11b44c324064f2c984d2b9f474249b3 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/a11b44c324064f2c984d2b9f474249b3 2023-07-15 18:13:12,309 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/a11b44c324064f2c984d2b9f474249b3, entries=77, sequenceid=32, filesize=14.1 K 2023-07-15 18:13:12,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/table/18ae753c949e461f952f90d512445679 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table/18ae753c949e461f952f90d512445679 2023-07-15 18:13:12,319 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table/18ae753c949e461f952f90d512445679, entries=6, sequenceid=32, filesize=4.9 K 2023-07-15 18:13:12,320 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~9.45 KB/9672, heapSize ~16.05 KB/16440, currentSize=0 B/0 for 1588230740 in 78ms, sequenceid=32, compaction requested=false; wal=null 2023-07-15 18:13:12,322 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/0000000000000000032 2023-07-15 18:13:12,326 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 18:13:12,327 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 18:13:12,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/32.seqid, newMaxSeqId=32, maxSeqId=1 2023-07-15 18:13:12,332 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=33; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10054861280, jitterRate=-0.06356807053089142}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 18:13:12,332 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 18:13:12,333 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=38, masterSystemTime=1689444792181 2023-07-15 18:13:12,335 WARN [ReadOnlyZKClient-127.0.0.1:62148@0x0689ffaf] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 18:13:12,336 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 275 connection: 172.31.14.131:33790 deadline: 1689444852335, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,336 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 293 connection: 172.31.14.131:33790 deadline: 1689444852335, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,336 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 18:13:12,336 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 18:13:12,336 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33055,1689444781474, state=OPEN 2023-07-15 18:13:12,338 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 18:13:12,339 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:12,342 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=37 2023-07-15 18:13:12,343 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=37, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33055,1689444781474 in 311 msec 2023-07-15 18:13:12,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=32 2023-07-15 18:13:12,345 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,43763,1689444781009 had 2 regions 2023-07-15 18:13:12,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=32, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 320 msec 2023-07-15 18:13:12,346 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=32, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43763,1689444781009, splitWal=true, meta=true, isMeta: false 2023-07-15 18:13:12,348 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,43763,1689444781009 WAL count=1, meta=false 2023-07-15 18:13:12,348 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=32, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682}] 2023-07-15 18:13:12,350 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,351 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, worker=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:12,504 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33055] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=40 2023-07-15 18:13:12,507 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 18:13:12,507 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 18:13:12,512 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 18:13:12,512 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 18:13:12,521 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, size=1.0 K (1047bytes) 2023-07-15 18:13:12,521 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 2023-07-15 18:13:12,522 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 after 1ms 2023-07-15 18:13:12,523 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42063 this server is in the failed servers list 2023-07-15 18:13:12,525 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 took 4ms 2023-07-15 18:13:12,528 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 83c5639bc741d79ce2b4c69e5abc7617: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-15 18:13:12,529 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 516b5326c0ebf25df25947c8572ab42d: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 18446744073709551615 } 2023-07-15 18:13:12,529 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 so closing down 2023-07-15 18:13:12,529 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 18:13:12,530 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 18:13:12,530 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 3 edits across 0 Regions in 5 ms; skipped=3; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, size=1.0 K, length=1047, corrupted=false, cancelled=false 2023-07-15 18:13:12,530 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, journal: Splitting hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, size=1.0 K (1047bytes) at 1689444792521Finishing writing output for hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 so closing down at 1689444792529 (+8 ms)3 split writer threads finished at 1689444792530 (+1 ms)Processed 3 edits across 0 Regions in 5 ms; skipped=3; WAL=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, size=1.0 K, length=1047, corrupted=false, cancelled=false at 1689444792530 2023-07-15 18:13:12,530 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 2023-07-15 18:13:12,531 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=40 2023-07-15 18:13:12,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster(3641): Remote procedure done, pid=40 2023-07-15 18:13:12,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] wal.WALSplitUtil(141): Moved hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting/jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682 to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:12,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-07-15 18:13:12,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, worker=jenkins-hbase4.apache.org,33055,1689444781474 in 184 msec 2023-07-15 18:13:12,538 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=32 2023-07-15 18:13:12,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=32, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43763%2C1689444781009.1689444784682, worker=jenkins-hbase4.apache.org,33055,1689444781474 in 191 msec 2023-07-15 18:13:12,543 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43763,1689444781009-splitting dir is empty, no logs to split. 2023-07-15 18:13:12,543 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,43763,1689444781009 WAL count=0, meta=false 2023-07-15 18:13:12,543 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43763,1689444781009 WAL splitting is done? wals=0, meta=false 2023-07-15 18:13:12,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=32, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN}] 2023-07-15 18:13:12,546 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=32, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN 2023-07-15 18:13:12,546 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=32, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:12,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=35, state=RUNNABLE; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:12,642 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=36, state=RUNNABLE; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:12,697 DEBUG [jenkins-hbase4:42385] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 18:13:12,697 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,697 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444792697"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444792697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444792697"}]},"ts":"1689444792697"} 2023-07-15 18:13:12,700 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=41, state=RUNNABLE; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:12,730 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42063 this server is in the failed servers list 2023-07-15 18:13:12,797 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:12,797 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 994da785cf935856bc388ebe43136633, NAME => 'hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:12,798 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:12,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. service=MultiRowMutationService 2023-07-15 18:13:12,799 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:rsgroup successfully. 2023-07-15 18:13:12,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table rsgroup 994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:12,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,801 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family m of region 994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,802 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:12,802 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:12,803 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 994da785cf935856bc388ebe43136633 columnFamilyName m 2023-07-15 18:13:12,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:12,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:12,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table Group_testFallback 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:12,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,805 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,808 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:12,808 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:12,808 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 516b5326c0ebf25df25947c8572ab42d columnFamilyName f 2023-07-15 18:13:12,809 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(310): Store=516b5326c0ebf25df25947c8572ab42d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:12,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,812 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1 2023-07-15 18:13:12,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,812 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(310): Store=994da785cf935856bc388ebe43136633/m, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:12,813 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,815 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,816 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000019 2023-07-15 18:13:12,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:12,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 516b5326c0ebf25df25947c8572ab42d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=9421027520, jitterRate=-0.12259843945503235}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:12,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:12,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d., pid=43, masterSystemTime=1689444792793 2023-07-15 18:13:12,821 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 20, skipped 0, firstSequenceIdInLog=15, maxSequenceIdInLog=19, path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000019 2023-07-15 18:13:12,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:12,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:12,821 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 994da785cf935856bc388ebe43136633 1/1 column families, dataSize=1.95 KB heapSize=3.70 KB 2023-07-15 18:13:12,822 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,822 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444792822"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444792822"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444792822"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444792822"}]},"ts":"1689444792822"} 2023-07-15 18:13:12,829 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=36 2023-07-15 18:13:12,829 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=36, state=SUCCESS; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,33055,1689444781474 in 182 msec 2023-07-15 18:13:12,831 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=29, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, ASSIGN in 807 msec 2023-07-15 18:13:12,842 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.95 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/aece4d2030694e0397949a0796daaa0d 2023-07-15 18:13:12,850 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/aece4d2030694e0397949a0796daaa0d as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d 2023-07-15 18:13:12,858 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d, entries=20, sequenceid=19, filesize=6.7 K 2023-07-15 18:13:12,858 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.95 KB/1998, heapSize ~3.69 KB/3776, currentSize=0 B/0 for 994da785cf935856bc388ebe43136633 in 37ms, sequenceid=19, compaction requested=false; wal=null 2023-07-15 18:13:12,859 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/0000000000000000019 2023-07-15 18:13:12,862 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:12,865 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=12 2023-07-15 18:13:12,866 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 994da785cf935856bc388ebe43136633; next sequenceid=20; org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy@7accd3df, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:12,866 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:12,867 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633., pid=42, masterSystemTime=1689444792793 2023-07-15 18:13:12,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:12,869 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:12,869 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:12,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 83c5639bc741d79ce2b4c69e5abc7617, NAME => 'hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:12,870 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=35 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPEN, openSeqNum=20, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,870 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,870 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:12,870 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444792869"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444792869"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444792869"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444792869"}]},"ts":"1689444792869"} 2023-07-15 18:13:12,870 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,870 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,872 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=35 2023-07-15 18:13:12,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=35, state=SUCCESS; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474 in 232 msec 2023-07-15 18:13:12,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=29 2023-07-15 18:13:12,879 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,42063,1689444786274 after splitting done 2023-07-15 18:13:12,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=29, state=SUCCESS; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, ASSIGN in 855 msec 2023-07-15 18:13:12,879 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,42063,1689444786274 from processing; numProcessing=1 2023-07-15 18:13:12,881 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:12,881 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:12,882 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,42063,1689444786274, splitWal=true, meta=false in 1.0920 sec 2023-07-15 18:13:12,882 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 83c5639bc741d79ce2b4c69e5abc7617 columnFamilyName info 2023-07-15 18:13:12,892 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info/cdaf017d393d46f59089b5495bf62ca2 2023-07-15 18:13:12,892 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(310): Store=83c5639bc741d79ce2b4c69e5abc7617/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:12,893 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,896 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,897 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5234): Maximum sequenceid for this wal is 5 and minimum sequenceid for the region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. is 6, skipped the whole file, path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000005 2023-07-15 18:13:12,898 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/0000000000000000005 2023-07-15 18:13:12,902 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:12,903 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 83c5639bc741d79ce2b4c69e5abc7617; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10463086080, jitterRate=-0.02554917335510254}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:12,904 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:12,905 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617., pid=44, masterSystemTime=1689444792793 2023-07-15 18:13:12,907 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:12,907 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:12,907 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:12,908 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444792907"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444792907"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444792907"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444792907"}]},"ts":"1689444792907"} 2023-07-15 18:13:12,912 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=41 2023-07-15 18:13:12,912 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=41, state=SUCCESS; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,33055,1689444781474 in 210 msec 2023-07-15 18:13:12,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=32 2023-07-15 18:13:12,914 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,43763,1689444781009 after splitting done 2023-07-15 18:13:12,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=32, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, ASSIGN in 367 msec 2023-07-15 18:13:12,915 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,43763,1689444781009 from processing; numProcessing=0 2023-07-15 18:13:12,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,43763,1689444781009, splitWal=true, meta=true in 1.0320 sec 2023-07-15 18:13:12,963 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:12,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=fallback 2023-07-15 18:13:12,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:12,984 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:12,985 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:12,985 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:12,985 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:12,985 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:12,985 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:12,985 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:12,989 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38101 2023-07-15 18:13:12,990 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:12,991 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:12,992 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:12,994 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:12,996 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38101 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:13,000 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:381010x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:13,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:38101-0x1016a3001d0000d connected 2023-07-15 18:13:13,002 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 18:13:13,003 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 18:13:13,004 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:13,006 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38101 2023-07-15 18:13:13,010 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38101 2023-07-15 18:13:13,013 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38101 2023-07-15 18:13:13,016 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38101 2023-07-15 18:13:13,016 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38101 2023-07-15 18:13:13,019 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:13,019 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:13,019 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:13,020 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:13,020 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:13,021 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:13,021 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:13,022 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 38111 2023-07-15 18:13:13,022 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:13,031 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:13,032 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@3a1f3aa1{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:13,033 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:13,033 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@1558c6f4{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:13,041 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:13,042 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:13,042 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:13,043 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 600000ms 2023-07-15 18:13:13,044 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:13,045 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@6388c156{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:13,047 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:13,047 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@71fc59fd{HTTP/1.1, (http/1.1)}{0.0.0.0:38111} 2023-07-15 18:13:13,048 INFO [Listener at localhost/46109] server.Server(415): Started @22058ms 2023-07-15 18:13:13,048 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:13,050 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:13,050 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:13,050 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:13,050 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:13,052 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:13,053 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 1 2023-07-15 18:13:13,054 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:13,055 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:13,055 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:13,058 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:13,058 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:13,059 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:13,059 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:13,060 DEBUG [RS:4;jenkins-hbase4:38101] zookeeper.ReadOnlyZKClient(139): Connect 0x0d9cfb61 to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:13,060 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:13,061 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:13,064 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 0 2023-07-15 18:13:13,067 DEBUG [RS:4;jenkins-hbase4:38101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35b14a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:13,067 DEBUG [RS:4;jenkins-hbase4:38101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7649011, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:13,077 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:4;jenkins-hbase4:38101 2023-07-15 18:13:13,077 INFO [RS:4;jenkins-hbase4:38101] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:13,077 INFO [RS:4;jenkins-hbase4:38101] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:13,077 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:13,078 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:38101, startcode=1689444792984 2023-07-15 18:13:13,078 DEBUG [RS:4;jenkins-hbase4:38101] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:13,081 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59203, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:13,082 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,082 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:13,082 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:13,082 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:13,082 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:13,084 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:13,084 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:13,084 DEBUG [RS:4;jenkins-hbase4:38101] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,084 WARN [RS:4;jenkins-hbase4:38101] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:13,084 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38101,1689444792984] 2023-07-15 18:13:13,084 INFO [RS:4;jenkins-hbase4:38101] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:13,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,085 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,085 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:13,086 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:13,086 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:13,086 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:13,087 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:13,090 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 1 2023-07-15 18:13:13,091 DEBUG [RS:4;jenkins-hbase4:38101] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,092 DEBUG [RS:4;jenkins-hbase4:38101] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,093 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:13,093 INFO [RS:4;jenkins-hbase4:38101] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:13,095 INFO [RS:4;jenkins-hbase4:38101] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:13,095 INFO [RS:4;jenkins-hbase4:38101] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:13,096 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:13,096 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:13,097 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:13,097 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,097 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,098 DEBUG [RS:4;jenkins-hbase4:38101] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:13,102 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:13,102 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:13,103 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:13,118 INFO [RS:4;jenkins-hbase4:38101] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:13,118 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38101,1689444792984-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:13,130 INFO [RS:4;jenkins-hbase4:38101] regionserver.Replication(203): jenkins-hbase4.apache.org,38101,1689444792984 started 2023-07-15 18:13:13,130 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,38101,1689444792984, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38101, sessionid=0x1016a3001d0000d 2023-07-15 18:13:13,130 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:13,131 DEBUG [RS:4;jenkins-hbase4:38101] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,131 DEBUG [RS:4;jenkins-hbase4:38101] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38101,1689444792984' 2023-07-15 18:13:13,131 DEBUG [RS:4;jenkins-hbase4:38101] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:13,131 DEBUG [RS:4;jenkins-hbase4:38101] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:13,132 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:13,132 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:13,132 DEBUG [RS:4;jenkins-hbase4:38101] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,132 DEBUG [RS:4;jenkins-hbase4:38101] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38101,1689444792984' 2023-07-15 18:13:13,132 DEBUG [RS:4;jenkins-hbase4:38101] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:13,132 DEBUG [RS:4;jenkins-hbase4:38101] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:13,133 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [3,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:13,133 DEBUG [RS:4;jenkins-hbase4:38101] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:13,133 INFO [RS:4;jenkins-hbase4:38101] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:13,133 INFO [RS:4;jenkins-hbase4:38101] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:13,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:13,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:13,137 INFO [Listener at localhost/46109] rsgroup.RSGroupBasedLoadBalancer(150): Start Generate Balance plan for group: default 2023-07-15 18:13:13,137 DEBUG [Listener at localhost/46109] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-15 18:13:13,137 INFO [Listener at localhost/46109] rsgroup.RSGroupBasedLoadBalancer(150): Start Generate Balance plan for group: Group_testFallback_1624499712 2023-07-15 18:13:13,138 DEBUG [Listener at localhost/46109] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-15 18:13:13,138 INFO [Listener at localhost/46109] master.HMaster(1846): Balancer plans size is 4, the balance interval is 0 ms, and the max number regions in transition is 4 2023-07-15 18:13:13,138 INFO [Listener at localhost/46109] master.HMaster(1851): balance hri=1588230740, source=jenkins-hbase4.apache.org,33055,1689444781474, destination= 2023-07-15 18:13:13,139 DEBUG [Listener at localhost/46109] procedure2.ProcedureExecutor(1029): Stored pid=45, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE 2023-07-15 18:13:13,139 INFO [Listener at localhost/46109] master.HMaster(1851): balance hri=83c5639bc741d79ce2b4c69e5abc7617, source=jenkins-hbase4.apache.org,33055,1689444781474, destination= 2023-07-15 18:13:13,140 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE 2023-07-15 18:13:13,141 DEBUG [Listener at localhost/46109] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE 2023-07-15 18:13:13,141 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33055,1689444781474, state=CLOSING 2023-07-15 18:13:13,141 INFO [Listener at localhost/46109] master.HMaster(1851): balance hri=994da785cf935856bc388ebe43136633, source=jenkins-hbase4.apache.org,33055,1689444781474, destination= 2023-07-15 18:13:13,142 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=46, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE 2023-07-15 18:13:13,143 DEBUG [Listener at localhost/46109] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE 2023-07-15 18:13:13,143 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=46 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,143 INFO [Listener at localhost/46109] master.HMaster(1851): balance hri=516b5326c0ebf25df25947c8572ab42d, source=jenkins-hbase4.apache.org,33055,1689444781474, destination= 2023-07-15 18:13:13,144 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE 2023-07-15 18:13:13,144 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444793143"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444793143"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444793143"}]},"ts":"1689444793143"} 2023-07-15 18:13:13,145 DEBUG [Listener at localhost/46109] procedure2.ProcedureExecutor(1029): Stored pid=48, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE 2023-07-15 18:13:13,146 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,146 DEBUG [Listener at localhost/46109] master.HMaster(1882): Balancer is going into sleep until next period in 300000ms 2023-07-15 18:13:13,146 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE 2023-07-15 18:13:13,146 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444793145"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444793145"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444793145"}]},"ts":"1689444793145"} 2023-07-15 18:13:13,147 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=45, state=RUNNABLE; CloseRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:13,147 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 18:13:13,147 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:13,149 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,149 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=46, state=RUNNABLE; CloseRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:13,149 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444793149"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444793149"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444793149"}]},"ts":"1689444793149"} 2023-07-15 18:13:13,149 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=47, state=RUNNABLE; CloseRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:13,150 DEBUG [PEWorker-4] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=50, ppid=46, state=RUNNABLE; CloseRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,151 DEBUG [PEWorker-2] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=51, ppid=47, state=RUNNABLE; CloseRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=48, state=RUNNABLE; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,33055,1689444781474}] 2023-07-15 18:13:13,153 DEBUG [PEWorker-4] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=52, ppid=48, state=RUNNABLE; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,235 INFO [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38101%2C1689444792984, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,38101,1689444792984, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:13,254 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:13,254 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:13,254 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:13,257 INFO [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,38101,1689444792984/jenkins-hbase4.apache.org%2C38101%2C1689444792984.1689444793236 2023-07-15 18:13:13,258 DEBUG [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:13,301 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 1588230740 2023-07-15 18:13:13,302 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 18:13:13,302 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 18:13:13,302 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 18:13:13,302 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 18:13:13,303 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 18:13:13,303 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.94 KB heapSize=7.13 KB 2023-07-15 18:13:13,319 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.94 KB at sequenceid=44 (bloomFilter=false), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/info/d3aa3bf21ece464e854a90cb050ad79c 2023-07-15 18:13:13,328 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/info/d3aa3bf21ece464e854a90cb050ad79c as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/d3aa3bf21ece464e854a90cb050ad79c 2023-07-15 18:13:13,336 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/d3aa3bf21ece464e854a90cb050ad79c, entries=33, sequenceid=44, filesize=8.6 K 2023-07-15 18:13:13,338 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.94 KB/4034, heapSize ~6.61 KB/6768, currentSize=0 B/0 for 1588230740 in 35ms, sequenceid=44, compaction requested=false 2023-07-15 18:13:13,351 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/47.seqid, newMaxSeqId=47, maxSeqId=32 2023-07-15 18:13:13,352 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 18:13:13,352 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 18:13:13,352 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 18:13:13,355 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 1588230740 2023-07-15 18:13:13,356 WARN [PEWorker-2] zookeeper.MetaTableLocator(225): Tried to set null ServerName in hbase:meta; skipping -- ServerName required 2023-07-15 18:13:13,359 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=45 2023-07-15 18:13:13,359 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=45, state=SUCCESS; CloseRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33055,1689444781474 in 209 msec 2023-07-15 18:13:13,361 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=45, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE; state=CLOSED, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:13,511 DEBUG [jenkins-hbase4:42385] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 18:13:13,512 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38101,1689444792984, state=OPENING 2023-07-15 18:13:13,513 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 18:13:13,513 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=53, ppid=45, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38101,1689444792984}] 2023-07-15 18:13:13,513 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:13,667 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,667 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 18:13:13,668 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 18:13:13,672 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 18:13:13,673 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:13,674 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38101%2C1689444792984.meta, suffix=.meta, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,38101,1689444792984, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:13,692 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:13,692 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:13,693 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:13,699 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,38101,1689444792984/jenkins-hbase4.apache.org%2C38101%2C1689444792984.meta.1689444793675.meta 2023-07-15 18:13:13,700 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:13,700 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:13,700 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:13,700 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 18:13:13,700 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 18:13:13,700 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 18:13:13,701 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:13,701 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 18:13:13,701 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 18:13:13,703 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 18:13:13,704 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:13,705 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info 2023-07-15 18:13:13,705 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 18:13:13,714 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/a11b44c324064f2c984d2b9f474249b3 2023-07-15 18:13:13,719 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/d3aa3bf21ece464e854a90cb050ad79c 2023-07-15 18:13:13,720 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:13,720 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 18:13:13,721 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:13,721 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier 2023-07-15 18:13:13,721 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 18:13:13,722 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:13,722 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 18:13:13,723 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:13,723 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table 2023-07-15 18:13:13,723 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 18:13:13,729 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table/18ae753c949e461f952f90d512445679 2023-07-15 18:13:13,730 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:13,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:13,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740 2023-07-15 18:13:13,734 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 18:13:13,736 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 18:13:13,737 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=48; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=10364776960, jitterRate=-0.03470492362976074}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 18:13:13,737 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 18:13:13,738 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=53, masterSystemTime=1689444793667 2023-07-15 18:13:13,742 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 18:13:13,742 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 18:13:13,743 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38101,1689444792984, state=OPEN 2023-07-15 18:13:13,744 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 18:13:13,744 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 18:13:13,747 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=53, resume processing ppid=45 2023-07-15 18:13:13,748 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=45, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38101,1689444792984 in 231 msec 2023-07-15 18:13:13,749 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE in 609 msec 2023-07-15 18:13:13,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 994da785cf935856bc388ebe43136633 2023-07-15 18:13:13,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 994da785cf935856bc388ebe43136633, disabling compactions & flushes 2023-07-15 18:13:13,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:13,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:13,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. after waiting 0 ms 2023-07-15 18:13:13,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:13,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 994da785cf935856bc388ebe43136633 1/1 column families, dataSize=1.08 KB heapSize=2.23 KB 2023-07-15 18:13:13,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.08 KB at sequenceid=25 (bloomFilter=true), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/abdb8e4d3709431ea2ab44d8c4e1c40f 2023-07-15 18:13:13,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/abdb8e4d3709431ea2ab44d8c4e1c40f as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f 2023-07-15 18:13:13,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f, entries=4, sequenceid=25, filesize=5.1 K 2023-07-15 18:13:13,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.08 KB/1104, heapSize ~2.22 KB/2272, currentSize=0 B/0 for 994da785cf935856bc388ebe43136633 in 35ms, sequenceid=25, compaction requested=true 2023-07-15 18:13:13,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/28.seqid, newMaxSeqId=28, maxSeqId=19 2023-07-15 18:13:13,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 18:13:13,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:13,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:13,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 994da785cf935856bc388ebe43136633 2023-07-15 18:13:13,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:13,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 516b5326c0ebf25df25947c8572ab42d, disabling compactions & flushes 2023-07-15 18:13:13,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:13,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:13,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. after waiting 0 ms 2023-07-15 18:13:13,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:13,945 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=CLOSED 2023-07-15 18:13:13,946 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444793945"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444793945"}]},"ts":"1689444793945"} 2023-07-15 18:13:13,946 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33055] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 213 connection: 172.31.14.131:33790 deadline: 1689444853946, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:13,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-15 18:13:13,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:13,950 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:13,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:13,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:13,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 83c5639bc741d79ce2b4c69e5abc7617, disabling compactions & flushes 2023-07-15 18:13:13,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:13,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:13,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. after waiting 0 ms 2023-07-15 18:13:13,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:13,952 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSED 2023-07-15 18:13:13,953 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444793952"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444793952"}]},"ts":"1689444793952"} 2023-07-15 18:13:13,955 DEBUG [PEWorker-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:13,956 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:13,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-07-15 18:13:13,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:13,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:13,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=48 2023-07-15 18:13:13,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=48, state=SUCCESS; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,33055,1689444781474 in 807 msec 2023-07-15 18:13:13,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:13,962 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=46 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=CLOSED 2023-07-15 18:13:13,962 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE; state=CLOSED, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:13,962 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444793962"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444793962"}]},"ts":"1689444793962"} 2023-07-15 18:13:13,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=46 2023-07-15 18:13:13,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=46, state=SUCCESS; CloseRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,33055,1689444781474 in 815 msec 2023-07-15 18:13:13,967 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=46, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE; state=CLOSED, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:13,967 DEBUG [jenkins-hbase4:42385] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 18:13:13,968 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=46 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,968 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:13,968 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444793968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444793968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444793968"}]},"ts":"1689444793968"} 2023-07-15 18:13:13,968 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444793968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444793968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444793968"}]},"ts":"1689444793968"} 2023-07-15 18:13:13,970 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=54, ppid=46, state=RUNNABLE; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,38101,1689444792984}] 2023-07-15 18:13:13,971 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=48, state=RUNNABLE; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,38101,1689444792984}] 2023-07-15 18:13:14,051 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=47 2023-07-15 18:13:14,052 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=47, state=SUCCESS; CloseRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,33055,1689444781474 in 900 msec 2023-07-15 18:13:14,052 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE; state=CLOSED, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:14,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:14,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:14,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table Group_testFallback 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:14,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,129 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,130 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:14,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 83c5639bc741d79ce2b4c69e5abc7617, NAME => 'hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:14,130 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:14,130 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:14,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:14,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,130 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 516b5326c0ebf25df25947c8572ab42d columnFamilyName f 2023-07-15 18:13:14,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,131 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(310): Store=516b5326c0ebf25df25947c8572ab42d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:14,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,132 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,134 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:14,134 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info 2023-07-15 18:13:14,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,134 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 83c5639bc741d79ce2b4c69e5abc7617 columnFamilyName info 2023-07-15 18:13:14,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:14,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 516b5326c0ebf25df25947c8572ab42d; next sequenceid=8; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=9514303680, jitterRate=-0.11391142010688782}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:14,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:14,143 DEBUG [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/info/cdaf017d393d46f59089b5495bf62ca2 2023-07-15 18:13:14,144 INFO [StoreOpener-83c5639bc741d79ce2b4c69e5abc7617-1] regionserver.HStore(310): Store=83c5639bc741d79ce2b4c69e5abc7617/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:14,144 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d., pid=55, masterSystemTime=1689444794122 2023-07-15 18:13:14,146 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:14,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:14,149 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPEN, openSeqNum=8, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,149 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444794149"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444794149"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444794149"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444794149"}]},"ts":"1689444794149"} 2023-07-15 18:13:14,151 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:14,153 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 83c5639bc741d79ce2b4c69e5abc7617; next sequenceid=13; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=11072132800, jitterRate=0.031172722578048706}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:14,153 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:14,154 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=48 2023-07-15 18:13:14,154 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=48, state=SUCCESS; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,38101,1689444792984 in 180 msec 2023-07-15 18:13:14,155 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617., pid=54, masterSystemTime=1689444794122 2023-07-15 18:13:14,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE in 1.0100 sec 2023-07-15 18:13:14,157 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:14,157 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:14,158 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=46 updating hbase:meta row=83c5639bc741d79ce2b4c69e5abc7617, regionState=OPEN, openSeqNum=13, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,158 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689444794157"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444794157"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444794157"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444794157"}]},"ts":"1689444794157"} 2023-07-15 18:13:14,162 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=54, resume processing ppid=46 2023-07-15 18:13:14,162 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=46, state=SUCCESS; OpenRegionProcedure 83c5639bc741d79ce2b4c69e5abc7617, server=jenkins-hbase4.apache.org,38101,1689444792984 in 190 msec 2023-07-15 18:13:14,164 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=83c5639bc741d79ce2b4c69e5abc7617, REOPEN/MOVE in 1.0220 sec 2023-07-15 18:13:14,202 DEBUG [jenkins-hbase4:42385] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 18:13:14,203 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,203 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444794203"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444794203"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444794203"}]},"ts":"1689444794203"} 2023-07-15 18:13:14,205 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=47, state=RUNNABLE; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,38101,1689444792984}] 2023-07-15 18:13:14,362 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:14,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 994da785cf935856bc388ebe43136633, NAME => 'hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:14,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 18:13:14,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. service=MultiRowMutationService 2023-07-15 18:13:14,363 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:rsgroup successfully. 2023-07-15 18:13:14,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table rsgroup 994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:14,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,365 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family m of region 994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,366 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:14,366 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m 2023-07-15 18:13:14,366 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 994da785cf935856bc388ebe43136633 columnFamilyName m 2023-07-15 18:13:14,376 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1 2023-07-15 18:13:14,382 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f 2023-07-15 18:13:14,387 DEBUG [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(539): loaded hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d 2023-07-15 18:13:14,387 INFO [StoreOpener-994da785cf935856bc388ebe43136633-1] regionserver.HStore(310): Store=994da785cf935856bc388ebe43136633/m, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:14,388 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,390 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,393 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:14,395 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 994da785cf935856bc388ebe43136633; next sequenceid=29; org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy@4168e992, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:14,395 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:14,395 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633., pid=56, masterSystemTime=1689444794358 2023-07-15 18:13:14,398 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-07-15 18:13:14,400 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-07-15 18:13:14,401 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:14,401 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:14,402 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=994da785cf935856bc388ebe43136633, regionState=OPEN, openSeqNum=29, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,402 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.","families":{"info":[{"qualifier":"regioninfo","vlen":39,"tag":[],"timestamp":"1689444794402"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444794402"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444794402"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444794402"}]},"ts":"1689444794402"} 2023-07-15 18:13:14,403 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 17329 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-07-15 18:13:14,405 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.HStore(1912): 994da785cf935856bc388ebe43136633/m is initiating minor compaction (all files) 2023-07-15 18:13:14,405 INFO [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 994da785cf935856bc388ebe43136633/m in hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:14,406 INFO [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f] into tmpdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp, totalSize=16.9 K 2023-07-15 18:13:14,406 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=47 2023-07-15 18:13:14,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=47, state=SUCCESS; OpenRegionProcedure 994da785cf935856bc388ebe43136633, server=jenkins-hbase4.apache.org,38101,1689444792984 in 199 msec 2023-07-15 18:13:14,407 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] compactions.Compactor(207): Compacting 1cca1b552a7046428556b030adf432e1, keycount=3, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1689444786653 2023-07-15 18:13:14,407 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] compactions.Compactor(207): Compacting aece4d2030694e0397949a0796daaa0d, keycount=20, bloomtype=ROW, size=6.7 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1689444787937 2023-07-15 18:13:14,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; TransitRegionStateProcedure table=hbase:rsgroup, region=994da785cf935856bc388ebe43136633, REOPEN/MOVE in 1.2640 sec 2023-07-15 18:13:14,408 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] compactions.Compactor(207): Compacting abdb8e4d3709431ea2ab44d8c4e1c40f, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=25, earliestPutTs=1689444793083 2023-07-15 18:13:14,425 INFO [RS:4;jenkins-hbase4:38101-shortCompactions-0] throttle.PressureAwareThroughputController(145): 994da785cf935856bc388ebe43136633#m#compaction#7 average throughput is 0.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-15 18:13:14,450 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/a18f87389d8c4193b791decab90fb80c as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/a18f87389d8c4193b791decab90fb80c 2023-07-15 18:13:14,466 INFO [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 994da785cf935856bc388ebe43136633/m of 994da785cf935856bc388ebe43136633 into a18f87389d8c4193b791decab90fb80c(size=5.3 K), total size for store is 5.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-15 18:13:14,466 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:14,466 INFO [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633., storeName=994da785cf935856bc388ebe43136633/m, priority=13, startTime=1689444794397; duration=0sec 2023-07-15 18:13:14,467 DEBUG [RS:4;jenkins-hbase4:38101-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-07-15 18:13:14,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:14,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:14,664 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:14,665 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43487 2023-07-15 18:13:14,666 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:14,667 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:14,668 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:14,669 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:14,670 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43487 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:14,674 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:434870x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:14,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43487-0x1016a3001d0000f connected 2023-07-15 18:13:14,676 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 18:13:14,676 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 18:13:14,677 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:14,678 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43487 2023-07-15 18:13:14,678 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43487 2023-07-15 18:13:14,678 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43487 2023-07-15 18:13:14,679 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43487 2023-07-15 18:13:14,679 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43487 2023-07-15 18:13:14,682 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:14,682 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:14,682 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:14,682 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:14,682 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:14,683 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:14,683 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:14,684 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 43885 2023-07-15 18:13:14,684 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:14,685 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:14,685 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@51056a75{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:14,686 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:14,686 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@71820257{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:14,695 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:14,697 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:14,697 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:14,697 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 600000ms 2023-07-15 18:13:14,698 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:14,700 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@25a23d21{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:14,702 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@70fccade{HTTP/1.1, (http/1.1)}{0.0.0.0:43885} 2023-07-15 18:13:14,702 INFO [Listener at localhost/46109] server.Server(415): Started @23713ms 2023-07-15 18:13:14,705 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:14,706 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:14,709 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:14,709 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:14,711 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:14,714 DEBUG [RS:5;jenkins-hbase4:43487] zookeeper.ReadOnlyZKClient(139): Connect 0x13929160 to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:14,718 DEBUG [RS:5;jenkins-hbase4:43487] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17ad17ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:14,719 DEBUG [RS:5;jenkins-hbase4:43487] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52af9b5b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:14,727 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:5;jenkins-hbase4:43487 2023-07-15 18:13:14,728 INFO [RS:5;jenkins-hbase4:43487] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:14,728 INFO [RS:5;jenkins-hbase4:43487] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:14,728 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:14,728 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:43487, startcode=1689444794663 2023-07-15 18:13:14,728 DEBUG [RS:5;jenkins-hbase4:43487] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:14,731 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57387, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:14,731 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,731 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:14,732 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:14,732 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:14,732 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:14,732 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33055] ipc.CallRunner(144): callId: 122 service: ClientService methodName: ExecService size: 630 connection: 172.31.14.131:33790 deadline: 1689444854732, exception=org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=jenkins-hbase4.apache.org port=42063 startCode=1689444786274. As of locationSeqNum=9. 2023-07-15 18:13:14,733 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:14,733 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:14,733 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:14,734 DEBUG [RS:5;jenkins-hbase4:43487] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,734 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43487,1689444794663] 2023-07-15 18:13:14,734 WARN [RS:5;jenkins-hbase4:43487] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:14,734 INFO [RS:5;jenkins-hbase4:43487] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:14,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,734 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:14,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:14,740 DEBUG [RS:5;jenkins-hbase4:43487] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:14,740 DEBUG [RS:5;jenkins-hbase4:43487] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,741 DEBUG [RS:5;jenkins-hbase4:43487] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:14,741 DEBUG [RS:5;jenkins-hbase4:43487] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:14,742 INFO [RS:5;jenkins-hbase4:43487] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:14,744 INFO [RS:5;jenkins-hbase4:43487] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:14,744 INFO [RS:5;jenkins-hbase4:43487] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:14,744 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:14,744 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:14,746 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,746 DEBUG [RS:5;jenkins-hbase4:43487] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:14,747 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:14,747 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:14,747 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:14,759 INFO [RS:5;jenkins-hbase4:43487] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:14,759 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43487,1689444794663-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:14,770 INFO [RS:5;jenkins-hbase4:43487] regionserver.Replication(203): jenkins-hbase4.apache.org,43487,1689444794663 started 2023-07-15 18:13:14,770 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43487,1689444794663, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43487, sessionid=0x1016a3001d0000f 2023-07-15 18:13:14,771 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:14,771 DEBUG [RS:5;jenkins-hbase4:43487] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,771 DEBUG [RS:5;jenkins-hbase4:43487] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43487,1689444794663' 2023-07-15 18:13:14,771 DEBUG [RS:5;jenkins-hbase4:43487] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:14,771 DEBUG [RS:5;jenkins-hbase4:43487] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:14,772 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:14,772 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:14,772 DEBUG [RS:5;jenkins-hbase4:43487] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:14,772 DEBUG [RS:5;jenkins-hbase4:43487] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43487,1689444794663' 2023-07-15 18:13:14,772 DEBUG [RS:5;jenkins-hbase4:43487] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:14,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:43487] to rsgroup Group_testFallback_1624499712 2023-07-15 18:13:14,772 DEBUG [RS:5;jenkins-hbase4:43487] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:14,773 DEBUG [RS:5;jenkins-hbase4:43487] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:14,773 INFO [RS:5;jenkins-hbase4:43487] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:14,773 INFO [RS:5;jenkins-hbase4:43487] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:14,839 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42063 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42063 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:14,840 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42063 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42063 2023-07-15 18:13:14,875 INFO [RS:5;jenkins-hbase4:43487] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43487%2C1689444794663, suffix=, logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43487,1689444794663, archiveDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs, maxLogs=32 2023-07-15 18:13:14,894 DEBUG [RS-EventLoopGroup-9-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK] 2023-07-15 18:13:14,894 DEBUG [RS-EventLoopGroup-9-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK] 2023-07-15 18:13:14,895 DEBUG [RS-EventLoopGroup-9-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK] 2023-07-15 18:13:14,899 INFO [RS:5;jenkins-hbase4:43487] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,43487,1689444794663/jenkins-hbase4.apache.org%2C43487%2C1689444794663.1689444794876 2023-07-15 18:13:14,899 DEBUG [RS:5;jenkins-hbase4:43487] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37907,DS-b864938b-2b4e-42e3-bc03-41329159ae0d,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-bebda7c1-1962-40c0-a432-2ed88361e2ef,DISK], DatanodeInfoWithStorage[127.0.0.1:45437,DS-72fb6b70-73f8-4a70-b4bf-1e06c8d0f0cd,DISK]] 2023-07-15 18:13:15,054 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:15,055 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:15,056 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:15,056 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:15,056 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:15,060 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 2 2023-07-15 18:13:15,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:15,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:15,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:15,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:15,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:15,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(286): Moving 0 region(s) to group default, current retry=0 2023-07-15 18:13:15,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(261): All regions from [jenkins-hbase4.apache.org,43487,1689444794663] are moved back to default 2023-07-15 18:13:15,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(438): Move servers done: default => Group_testFallback_1624499712 2023-07-15 18:13:15,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveServers 2023-07-15 18:13:15,069 WARN [RPCClient-NioEventLoopGroup-6-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42063 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42063 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:15,070 DEBUG [RPCClient-NioEventLoopGroup-6-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42063 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42063 2023-07-15 18:13:15,171 WARN [RPCClient-NioEventLoopGroup-6-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43763 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43763 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:15,172 DEBUG [RPCClient-NioEventLoopGroup-6-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43763 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43763 2023-07-15 18:13:15,275 DEBUG [hconnection-0x5f48d716-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 18:13:15,277 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 18:13:15,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:15,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:15,296 INFO [Listener at localhost/46109] rsgroup.RSGroupBasedLoadBalancer(150): Start Generate Balance plan for group: default 2023-07-15 18:13:15,296 DEBUG [Listener at localhost/46109] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-15 18:13:15,296 INFO [Listener at localhost/46109] rsgroup.RSGroupBasedLoadBalancer(150): Start Generate Balance plan for group: Group_testFallback_1624499712 2023-07-15 18:13:15,296 DEBUG [Listener at localhost/46109] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-15 18:13:15,296 INFO [Listener at localhost/46109] master.HMaster(1846): Balancer plans size is 1, the balance interval is 0 ms, and the max number regions in transition is 4 2023-07-15 18:13:15,296 INFO [Listener at localhost/46109] master.HMaster(1851): balance hri=516b5326c0ebf25df25947c8572ab42d, source=jenkins-hbase4.apache.org,38101,1689444792984, destination= 2023-07-15 18:13:15,297 DEBUG [Listener at localhost/46109] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE 2023-07-15 18:13:15,297 DEBUG [Listener at localhost/46109] master.HMaster(1882): Balancer is going into sleep until next period in 300000ms 2023-07-15 18:13:15,299 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=57, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE 2023-07-15 18:13:15,300 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:15,300 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444795300"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444795300"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444795300"}]},"ts":"1689444795300"} 2023-07-15 18:13:15,302 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,38101,1689444792984}] 2023-07-15 18:13:15,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 516b5326c0ebf25df25947c8572ab42d, disabling compactions & flushes 2023-07-15 18:13:15,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. after waiting 0 ms 2023-07-15 18:13:15,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/10.seqid, newMaxSeqId=10, maxSeqId=7 2023-07-15 18:13:15,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:15,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,465 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSED 2023-07-15 18:13:15,465 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444795465"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444795465"}]},"ts":"1689444795465"} 2023-07-15 18:13:15,470 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-07-15 18:13:15,470 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,38101,1689444792984 in 166 msec 2023-07-15 18:13:15,471 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=57, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE; state=CLOSED, location=null; forceNewPlan=true, retain=false 2023-07-15 18:13:15,621 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:15,621 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444795621"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444795621"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444795621"}]},"ts":"1689444795621"} 2023-07-15 18:13:15,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=57, state=RUNNABLE; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43487,1689444794663}] 2023-07-15 18:13:15,696 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-15 18:13:15,787 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:15,788 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 18:13:15,792 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 18:13:15,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''} 2023-07-15 18:13:15,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table Group_testFallback 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 18:13:15,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,806 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,807 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:15,808 DEBUG [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f 2023-07-15 18:13:15,808 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 516b5326c0ebf25df25947c8572ab42d columnFamilyName f 2023-07-15 18:13:15,809 INFO [StoreOpener-516b5326c0ebf25df25947c8572ab42d-1] regionserver.HStore(310): Store=516b5326c0ebf25df25947c8572ab42d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 18:13:15,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:15,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 516b5326c0ebf25df25947c8572ab42d; next sequenceid=11; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=12073600960, jitterRate=0.1244417130947113}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 18:13:15,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:15,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d., pid=59, masterSystemTime=1689444795787 2023-07-15 18:13:15,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:15,820 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=OPEN, openSeqNum=11, regionLocation=jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:15,820 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444795820"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689444795820"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689444795820"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689444795820"}]},"ts":"1689444795820"} 2023-07-15 18:13:15,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=57 2023-07-15 18:13:15,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=57, state=SUCCESS; OpenRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43487,1689444794663 in 199 msec 2023-07-15 18:13:15,825 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, REOPEN/MOVE in 528 msec 2023-07-15 18:13:16,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=Group_testFallback_1624499712 2023-07-15 18:13:16,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:16,053 INFO [Listener at localhost/46109] client.HBaseAdmin$15(890): Started disable of Group_testFallback 2023-07-15 18:13:16,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable Group_testFallback 2023-07-15 18:13:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure2.ProcedureExecutor(1029): Stored pid=60, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=Group_testFallback 2023-07-15 18:13:16,069 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"Group_testFallback","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444796069"}]},"ts":"1689444796069"} 2023-07-15 18:13:16,071 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=Group_testFallback, state=DISABLING in hbase:meta 2023-07-15 18:13:16,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=60 2023-07-15 18:13:16,072 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set Group_testFallback to state=DISABLING 2023-07-15 18:13:16,074 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=60, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, UNASSIGN}] 2023-07-15 18:13:16,076 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=61, ppid=60, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, UNASSIGN 2023-07-15 18:13:16,077 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=61 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:16,077 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444796077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689444796077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689444796077"}]},"ts":"1689444796077"} 2023-07-15 18:13:16,078 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=62, ppid=61, state=RUNNABLE; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43487,1689444794663}] 2023-07-15 18:13:16,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=60 2023-07-15 18:13:16,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:16,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 516b5326c0ebf25df25947c8572ab42d, disabling compactions & flushes 2023-07-15 18:13:16,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:16,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:16,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. after waiting 0 ms 2023-07-15 18:13:16,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:16,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=10 2023-07-15 18:13:16,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d. 2023-07-15 18:13:16,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 516b5326c0ebf25df25947c8572ab42d: 2023-07-15 18:13:16,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:16,241 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=61 updating hbase:meta row=516b5326c0ebf25df25947c8572ab42d, regionState=CLOSED 2023-07-15 18:13:16,241 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689444796241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689444796241"}]},"ts":"1689444796241"} 2023-07-15 18:13:16,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=62, resume processing ppid=61 2023-07-15 18:13:16,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, ppid=61, state=SUCCESS; CloseRegionProcedure 516b5326c0ebf25df25947c8572ab42d, server=jenkins-hbase4.apache.org,43487,1689444794663 in 165 msec 2023-07-15 18:13:16,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=60 2023-07-15 18:13:16,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=60, state=SUCCESS; TransitRegionStateProcedure table=Group_testFallback, region=516b5326c0ebf25df25947c8572ab42d, UNASSIGN in 171 msec 2023-07-15 18:13:16,247 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"Group_testFallback","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689444796247"}]},"ts":"1689444796247"} 2023-07-15 18:13:16,249 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=Group_testFallback, state=DISABLED in hbase:meta 2023-07-15 18:13:16,250 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set Group_testFallback to state=DISABLED 2023-07-15 18:13:16,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, state=SUCCESS; DisableTableProcedure table=Group_testFallback in 191 msec 2023-07-15 18:13:16,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=60 2023-07-15 18:13:16,374 INFO [Listener at localhost/46109] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:Group_testFallback, procId: 60 completed 2023-07-15 18:13:16,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete Group_testFallback 2023-07-15 18:13:16,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=Group_testFallback 2023-07-15 18:13:16,554 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=63, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=Group_testFallback 2023-07-15 18:13:16,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint(577): Removing deleted table 'Group_testFallback' from rsgroup 'Group_testFallback_1624499712' 2023-07-15 18:13:16,556 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=63, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=Group_testFallback 2023-07-15 18:13:16,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:16,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:16,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:16,563 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=63 2023-07-15 18:13:16,566 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/f, FileablePath, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits] 2023-07-15 18:13:16,572 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/13.seqid to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/archive/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d/recovered.edits/13.seqid 2023-07-15 18:13:16,572 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/.tmp/data/default/Group_testFallback/516b5326c0ebf25df25947c8572ab42d 2023-07-15 18:13:16,572 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived Group_testFallback regions 2023-07-15 18:13:16,575 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=63, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=Group_testFallback 2023-07-15 18:13:16,600 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of Group_testFallback from hbase:meta 2023-07-15 18:13:16,645 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'Group_testFallback' descriptor. 2023-07-15 18:13:16,647 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=63, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=Group_testFallback 2023-07-15 18:13:16,647 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'Group_testFallback' from region states. 2023-07-15 18:13:16,647 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689444796647"}]},"ts":"9223372036854775807"} 2023-07-15 18:13:16,650 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-15 18:13:16,650 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 516b5326c0ebf25df25947c8572ab42d, NAME => 'Group_testFallback,,1689444789022.516b5326c0ebf25df25947c8572ab42d.', STARTKEY => '', ENDKEY => ''}] 2023-07-15 18:13:16,650 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'Group_testFallback' as deleted. 2023-07-15 18:13:16,650 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"Group_testFallback","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689444796650"}]},"ts":"9223372036854775807"} 2023-07-15 18:13:16,653 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table Group_testFallback state from META 2023-07-15 18:13:16,656 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=63, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=Group_testFallback 2023-07-15 18:13:16,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; DeleteTableProcedure table=Group_testFallback in 275 msec 2023-07-15 18:13:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(1230): Checking to see if procedure is done pid=63 2023-07-15 18:13:16,667 INFO [Listener at localhost/46109] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:Group_testFallback, procId: 63 completed 2023-07-15 18:13:16,669 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39349 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39349 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 18:13:16,671 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39349 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39349 2023-07-15 18:13:16,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:16,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:16,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(231): Client=jenkins//172.31.14.131 move tables [] to rsgroup default 2023-07-15 18:13:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(448): moveTables() passed an empty set. Ignoring. 2023-07-15 18:13:16,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveTables 2023-07-15 18:13:16,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:39349, jenkins-hbase4.apache.org:43487] to rsgroup default 2023-07-15 18:13:16,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(258): Dropping jenkins-hbase4.apache.org:39349 during move-to-default rsgroup because not online 2023-07-15 18:13:16,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/Group_testFallback_1624499712 2023-07-15 18:13:16,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:16,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:16,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 8 2023-07-15 18:13:16,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(286): Moving 0 region(s) to group Group_testFallback_1624499712, current retry=0 2023-07-15 18:13:16,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(261): All regions from [jenkins-hbase4.apache.org,43487,1689444794663] are moved back to Group_testFallback_1624499712 2023-07-15 18:13:16,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(438): Move servers done: Group_testFallback_1624499712 => default 2023-07-15 18:13:16,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveServers 2023-07-15 18:13:16,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(272): Client=jenkins//172.31.14.131 remove rsgroup Group_testFallback_1624499712 2023-07-15 18:13:16,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:16,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:16,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 7 2023-07-15 18:13:16,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.RemoveRSGroup 2023-07-15 18:13:16,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(231): Client=jenkins//172.31.14.131 move tables [] to rsgroup default 2023-07-15 18:13:16,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(448): moveTables() passed an empty set. Ignoring. 2023-07-15 18:13:16,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveTables 2023-07-15 18:13:16,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:33055] to rsgroup default 2023-07-15 18:13:16,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/fallback 2023-07-15 18:13:16,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:16,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 6 2023-07-15 18:13:16,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(286): Moving 0 region(s) to group fallback, current retry=0 2023-07-15 18:13:16,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(261): All regions from [jenkins-hbase4.apache.org,33055,1689444781474] are moved back to fallback 2023-07-15 18:13:16,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(438): Move servers done: fallback => default 2023-07-15 18:13:16,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveServers 2023-07-15 18:13:16,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(272): Client=jenkins//172.31.14.131 remove rsgroup fallback 2023-07-15 18:13:16,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:16,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 5 2023-07-15 18:13:16,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.RemoveRSGroup 2023-07-15 18:13:16,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(231): Client=jenkins//172.31.14.131 move tables [] to rsgroup default 2023-07-15 18:13:16,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminServer(448): moveTables() passed an empty set. Ignoring. 2023-07-15 18:13:16,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveTables 2023-07-15 18:13:16,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [] to rsgroup default 2023-07-15 18:13:16,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.MoveServers 2023-07-15 18:13:16,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(272): Client=jenkins//172.31.14.131 remove rsgroup master 2023-07-15 18:13:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 3 2023-07-15 18:13:16,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.RemoveRSGroup 2023-07-15 18:13:16,840 INFO [Listener at localhost/46109] rsgroup.TestRSGroupsBase(152): Restoring servers: 1 2023-07-15 18:13:16,852 INFO [Listener at localhost/46109] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 18:13:16,853 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:16,854 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:16,854 INFO [Listener at localhost/46109] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 18:13:16,854 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 18:13:16,854 INFO [Listener at localhost/46109] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 18:13:16,854 INFO [Listener at localhost/46109] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 18:13:16,855 INFO [Listener at localhost/46109] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44337 2023-07-15 18:13:16,855 INFO [Listener at localhost/46109] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 18:13:16,857 DEBUG [Listener at localhost/46109] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 18:13:16,857 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:16,859 INFO [Listener at localhost/46109] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 18:13:16,860 INFO [Listener at localhost/46109] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44337 connecting to ZooKeeper ensemble=127.0.0.1:62148 2023-07-15 18:13:16,865 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:443370x0, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 18:13:16,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:44337-0x1016a3001d00011 connected 2023-07-15 18:13:16,867 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 18:13:16,868 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 18:13:16,869 DEBUG [Listener at localhost/46109] zookeeper.ZKUtil(164): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 18:13:16,869 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44337 2023-07-15 18:13:16,869 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44337 2023-07-15 18:13:16,870 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44337 2023-07-15 18:13:16,870 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44337 2023-07-15 18:13:16,871 DEBUG [Listener at localhost/46109] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44337 2023-07-15 18:13:16,873 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2023-07-15 18:13:16,873 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2023-07-15 18:13:16,873 INFO [Listener at localhost/46109] http.HttpServer(900): Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2023-07-15 18:13:16,873 INFO [Listener at localhost/46109] http.HttpServer(879): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2023-07-15 18:13:16,873 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2023-07-15 18:13:16,874 INFO [Listener at localhost/46109] http.HttpServer(886): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-07-15 18:13:16,874 INFO [Listener at localhost/46109] http.HttpServer(783): ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2023-07-15 18:13:16,874 INFO [Listener at localhost/46109] http.HttpServer(1146): Jetty bound to port 45837 2023-07-15 18:13:16,874 INFO [Listener at localhost/46109] server.Server(375): jetty-9.4.50.v20221201; built: 2022-12-01T22:07:03.915Z; git: da9a0b30691a45daf90a9f17b5defa2f1434f882; jvm 1.8.0_362-b09 2023-07-15 18:13:16,876 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:16,876 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@37a18b85{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,AVAILABLE} 2023-07-15 18:13:16,876 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:16,876 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.s.ServletContextHandler@5b71a7c8{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,AVAILABLE} 2023-07-15 18:13:16,882 INFO [Listener at localhost/46109] webapp.StandardDescriptorProcessor(277): NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2023-07-15 18:13:16,883 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(334): DefaultSessionIdManager workerName=node0 2023-07-15 18:13:16,883 INFO [Listener at localhost/46109] session.DefaultSessionIdManager(339): No SessionScavenger set, using defaults 2023-07-15 18:13:16,884 INFO [Listener at localhost/46109] session.HouseKeeper(132): node0 Scavenging every 600000ms 2023-07-15 18:13:16,884 INFO [Listener at localhost/46109] http.SecurityHeadersFilter(48): Added security headers filter 2023-07-15 18:13:16,885 INFO [Listener at localhost/46109] handler.ContextHandler(921): Started o.a.h.t.o.e.j.w.WebAppContext@549741d2{regionserver,/,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver/,AVAILABLE}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:16,887 INFO [Listener at localhost/46109] server.AbstractConnector(333): Started ServerConnector@d52b758{HTTP/1.1, (http/1.1)}{0.0.0.0:45837} 2023-07-15 18:13:16,887 INFO [Listener at localhost/46109] server.Server(415): Started @25898ms 2023-07-15 18:13:16,891 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(951): ClusterId : 1d505113-7488-4c3d-b421-4532e044e8dc 2023-07-15 18:13:16,894 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 18:13:16,896 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 18:13:16,896 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 18:13:16,898 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 18:13:16,899 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ReadOnlyZKClient(139): Connect 0x1422981d to 127.0.0.1:62148 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 18:13:16,904 DEBUG [RS:6;jenkins-hbase4:44337] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1edaf36d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2023-07-15 18:13:16,904 DEBUG [RS:6;jenkins-hbase4:44337] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5bd87ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:16,913 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:6;jenkins-hbase4:44337 2023-07-15 18:13:16,913 INFO [RS:6;jenkins-hbase4:44337] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 18:13:16,914 INFO [RS:6;jenkins-hbase4:44337] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 18:13:16,914 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 18:13:16,914 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42385,1689444778672 with isa=jenkins-hbase4.apache.org/172.31.14.131:44337, startcode=1689444796851 2023-07-15 18:13:16,914 DEBUG [RS:6;jenkins-hbase4:44337] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 18:13:16,917 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41327, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 18:13:16,917 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42385] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,917 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(787): Updating default servers. 2023-07-15 18:13:16,917 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117 2023-07-15 18:13:16,918 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39481 2023-07-15 18:13:16,918 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=41633 2023-07-15 18:13:16,922 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:16,923 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:16,922 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:16,923 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:16,923 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,923 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,923 WARN [RS:6;jenkins-hbase4:44337] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 18:13:16,923 INFO [RS:6;jenkins-hbase4:44337] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 18:13:16,923 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/WALs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,923 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44337,1689444796851] 2023-07-15 18:13:16,923 DEBUG [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 2 2023-07-15 18:13:16,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:16,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:16,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:16,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,925 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-jenkins-hbase4.apache.org,42385,1689444778672] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread(792): Updated with servers: 4 2023-07-15 18:13:16,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:16,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:16,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:16,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:16,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:16,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:16,929 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:16,929 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,929 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:16,930 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ZKUtil(162): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:16,930 DEBUG [RS:6;jenkins-hbase4:44337] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 18:13:16,931 INFO [RS:6;jenkins-hbase4:44337] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 18:13:16,932 INFO [RS:6;jenkins-hbase4:44337] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 18:13:16,932 INFO [RS:6;jenkins-hbase4:44337] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 18:13:16,932 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:16,932 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 18:13:16,934 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,934 DEBUG [RS:6;jenkins-hbase4:44337] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 18:13:16,935 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:16,935 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:16,935 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:16,947 INFO [RS:6;jenkins-hbase4:44337] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 18:13:16,947 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44337,1689444796851-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 18:13:16,958 INFO [RS:6;jenkins-hbase4:44337] regionserver.Replication(203): jenkins-hbase4.apache.org,44337,1689444796851 started 2023-07-15 18:13:16,958 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,44337,1689444796851, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44337, sessionid=0x1016a3001d00011 2023-07-15 18:13:16,958 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 18:13:16,958 DEBUG [RS:6;jenkins-hbase4:44337] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44337,1689444796851' 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:16,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(252): Client=jenkins//172.31.14.131 add rsgroup master 2023-07-15 18:13:16,959 DEBUG [RS:6;jenkins-hbase4:44337] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44337,1689444796851' 2023-07-15 18:13:16,960 DEBUG [RS:6;jenkins-hbase4:44337] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 18:13:16,960 DEBUG [RS:6;jenkins-hbase4:44337] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 18:13:16,960 DEBUG [RS:6;jenkins-hbase4:44337] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 18:13:16,960 INFO [RS:6;jenkins-hbase4:44337] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 18:13:16,960 INFO [RS:6;jenkins-hbase4:44337] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 18:13:16,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/default 2023-07-15 18:13:16,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(662): Updating znode: /hbase/rsgroup/master 2023-07-15 18:13:16,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupInfoManagerImpl(668): Writing ZK GroupInfo count: 4 2023-07-15 18:13:16,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.AddRSGroup 2023-07-15 18:13:16,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:16,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:16,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(206): Client=jenkins//172.31.14.131 move servers [jenkins-hbase4.apache.org:42385] to rsgroup master 2023-07-15 18:13:16,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:408) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:213) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16193) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:900) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 18:13:16,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] ipc.CallRunner(144): callId: 132 service: MasterService methodName: ExecMasterService size: 119 connection: 172.31.14.131:40862 deadline: 1689445996975, exception=org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. 2023-07-15 18:13:16,976 WARN [Listener at localhost/46109] rsgroup.TestRSGroupsBase(163): Got this on setup, FYI org.apache.hadoop.hbase.constraint.ConstraintException: org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:408) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:213) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16193) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:900) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101) at org.apache.hadoop.hbase.client.HBaseAdmin$74.callExecService(HBaseAdmin.java:2985) at org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:62) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService$BlockingStub.moveServers(RSGroupAdminProtos.java:16658) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient.moveServers(RSGroupAdminClient.java:108) at org.apache.hadoop.hbase.rsgroup.VerifyingRSGroupAdminClient.moveServers(VerifyingRSGroupAdminClient.java:77) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase.tearDownAfterMethod(TestRSGroupsBase.java:161) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsFallback.afterMethod(TestRSGroupsFallback.java:79) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.constraint.ConstraintException): org.apache.hadoop.hbase.constraint.ConstraintException: Server jenkins-hbase4.apache.org:42385 is either offline or it does not exist. at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:408) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:213) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:16193) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:900) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 18:13:16,978 INFO [Listener at localhost/46109] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 18:13:16,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(316): Client=jenkins//172.31.14.131 list rsgroup 2023-07-15 18:13:16,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.ListRSGroupInfos 2023-07-15 18:13:16,979 INFO [Listener at localhost/46109] rsgroup.TestRSGroupsBase$2(169): Waiting for cleanup to finish [Name:default, Servers:[jenkins-hbase4.apache.org:33055, jenkins-hbase4.apache.org:38101, jenkins-hbase4.apache.org:43487, jenkins-hbase4.apache.org:44337], Tables:[hbase:meta, hbase:namespace, hbase:rsgroup], Configurations:{}, Name:master, Servers:[], Tables:[], Configurations:{}] 2023-07-15 18:13:16,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl(165): Client=jenkins//172.31.14.131 initiates rsgroup info retrieval, group=default 2023-07-15 18:13:16,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] master.MasterRpcServices(912): User jenkins (auth:SIMPLE) (remote address: /172.31.14.131) master service request for RSGroupAdminService.GetRSGroupInfo 2023-07-15 18:13:17,015 INFO [Listener at localhost/46109] hbase.ResourceChecker(175): after: rsgroup.TestRSGroupsFallback#testFallback Thread=547 (was 425) Potentially hanging thread: Session-HouseKeeper-69cf061a-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1571326333_17 at /127.0.0.1:40590 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741848_1024] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp796451694-1079 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:39481 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp954283457-881 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:6;jenkins-hbase4:44337 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:42063Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x1422981d-SendThread(127.0.0.1:62148) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:58740 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp512752194-1002 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Abort regionserver monitor java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x0d9cfb61-SendThread(127.0.0.1:62148) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: qtp796451694-1078 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1850812016) connection to localhost/127.0.0.1:39481 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117-prefix:jenkins-hbase4.apache.org,33055,1689444781474.meta sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:5;jenkins-hbase4:43487 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp796451694-1082 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:44337Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1571326333_17 at /127.0.0.1:43962 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741848_1024] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f48d716-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x1422981d-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741854_1030, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:38101Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:44060 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741852_1028, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:43984 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741852_1028] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp512752194-1001 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x13929160-SendThread(127.0.0.1:62148) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:4;jenkins-hbase4:38101 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741857_1033, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f48d716-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp954283457-877 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:58632 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:6;jenkins-hbase4:44337-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:58672 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:58692 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741852_1028] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741852_1028, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x0d9cfb61-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Abort regionserver monitor java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117-prefix:jenkins-hbase4.apache.org,38101,1689444792984 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:44026 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:40676 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741854_1030] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp512752194-1000 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x13929160-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741854_1030, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:58720 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741852_1028, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741848_1024, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp796451694-1077-acceptor-0@ac091d8-ServerConnector@d52b758{HTTP/1.1, (http/1.1)}{0.0.0.0:45837} sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) org.apache.hbase.thirdparty.org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741857_1033, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1850812016) connection to localhost/127.0.0.1:39481 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:40662 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp796451694-1080 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp512752194-998 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741857_1033, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741854_1030, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:58722 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741854_1030] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:43487Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117-prefix:jenkins-hbase4.apache.org,43487,1689444794663 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:4;jenkins-hbase4:38101-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-metaLookup-shared--pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_10571783_17 at /127.0.0.1:44090 [Waiting for operation #20] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp954283457-882 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp796451694-1076 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$$Lambda$75/347835008.run(Unknown Source) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f48d716-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:40628 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp954283457-880 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_META-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Abort regionserver monitor java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:43976 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:58730 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1850812016) connection to localhost/127.0.0.1:39481 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: qtp796451694-1081 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp954283457-879 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Session-HouseKeeper-60b63831-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:55904 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:5;jenkins-hbase4:43487-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp512752194-997-acceptor-0@590749fc-ServerConnector@70fccade{HTTP/1.1, (http/1.1)}{0.0.0.0:43885} sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) org.apache.hbase.thirdparty.org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_10571783_17 at /127.0.0.1:58744 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741857_1033] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1850812016) connection to localhost/127.0.0.1:39481 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: Session-HouseKeeper-d597078-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741848_1024, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:39481 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:44006 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741854_1030] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1571326333_17 at /127.0.0.1:58644 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741848_1024] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: hconnection-0x3714f5c6-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp954283457-876-acceptor-0@17162742-ServerConnector@71fc59fd{HTTP/1.1, (http/1.1)}{0.0.0.0:38111} sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) org.apache.hbase.thirdparty.org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp512752194-996 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$$Lambda$75/347835008.run(Unknown Source) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp954283457-878 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=38101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-578561809-172.31.14.131-1689444773408:blk_1073741848_1024, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp512752194-999 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1477733836_17 at /127.0.0.1:44042 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44337 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: qtp512752194-1003 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x1422981d sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$89/1510732793.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x13929160 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$89/1510732793.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:39481 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:4;jenkins-hbase4:38101-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: qtp796451694-1083 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_624519512_17 at /127.0.0.1:40642 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741852_1028] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_10571783_17 at /127.0.0.1:40702 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741857_1033] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62148@0x0d9cfb61 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$89/1510732793.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3714f5c6-metaLookup-shared--pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=43487 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117-prefix:jenkins-hbase4.apache.org,38101,1689444792984.meta sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: qtp954283457-875 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$$Lambda$75/347835008.run(Unknown Source) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_10571783_17 at /127.0.0.1:44074 [Receiving block BP-578561809-172.31.14.131-1689444773408:blk_1073741857_1033] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=804 (was 679) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=723 (was 776), ProcessCount=177 (was 183), AvailableMemoryMB=2074 (was 1443) - AvailableMemoryMB LEAK? - 2023-07-15 18:13:17,017 WARN [Listener at localhost/46109] hbase.ResourceChecker(130): Thread=547 is superior to 500 2023-07-15 18:13:17,017 INFO [Listener at localhost/46109] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-15 18:13:17,018 INFO [Listener at localhost/46109] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-15 18:13:17,018 DEBUG [Listener at localhost/46109] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7075917a to 127.0.0.1:62148 2023-07-15 18:13:17,018 DEBUG [Listener at localhost/46109] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,018 DEBUG [Listener at localhost/46109] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-15 18:13:17,018 DEBUG [Listener at localhost/46109] util.JVMClusterUtil(257): Found active master hash=1069686756, stopped=false 2023-07-15 18:13:17,018 DEBUG [Listener at localhost/46109] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint 2023-07-15 18:13:17,018 DEBUG [Listener at localhost/46109] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver 2023-07-15 18:13:17,018 INFO [Listener at localhost/46109] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:17,020 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:17,020 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:17,020 INFO [Listener at localhost/46109] procedure2.ProcedureExecutor(629): Stopping 2023-07-15 18:13:17,020 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:17,020 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:17,020 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:17,020 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 18:13:17,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:17,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:17,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:17,021 DEBUG [Listener at localhost/46109] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0689ffaf to 127.0.0.1:62148 2023-07-15 18:13:17,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:17,021 DEBUG [Listener at localhost/46109] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 18:13:17,022 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,33055,1689444781474' ***** 2023-07-15 18:13:17,022 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 18:13:17,022 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,38101,1689444792984' ***** 2023-07-15 18:13:17,022 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 18:13:17,022 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:17,022 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:17,022 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43487,1689444794663' ***** 2023-07-15 18:13:17,023 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 18:13:17,025 INFO [Listener at localhost/46109] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,44337,1689444796851' ***** 2023-07-15 18:13:17,025 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:17,026 INFO [Listener at localhost/46109] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 18:13:17,029 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:17,035 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:17,036 INFO [RS:2;jenkins-hbase4:33055] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@41f3e3cf{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:17,036 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,040 INFO [RS:4;jenkins-hbase4:38101] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@6388c156{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:17,040 INFO [RS:6;jenkins-hbase4:44337] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@549741d2{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:17,041 INFO [RS:2;jenkins-hbase4:33055] server.AbstractConnector(383): Stopped ServerConnector@5922b370{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:17,040 INFO [RS:5;jenkins-hbase4:43487] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@25a23d21{regionserver,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/regionserver} 2023-07-15 18:13:17,041 INFO [RS:2;jenkins-hbase4:33055] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:17,041 INFO [RS:4;jenkins-hbase4:38101] server.AbstractConnector(383): Stopped ServerConnector@71fc59fd{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:17,041 INFO [RS:6;jenkins-hbase4:44337] server.AbstractConnector(383): Stopped ServerConnector@d52b758{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:17,042 INFO [RS:2;jenkins-hbase4:33055] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@10470979{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:17,042 INFO [RS:5;jenkins-hbase4:43487] server.AbstractConnector(383): Stopped ServerConnector@70fccade{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:17,042 INFO [RS:6;jenkins-hbase4:44337] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:17,041 INFO [RS:4;jenkins-hbase4:38101] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:17,042 INFO [RS:5;jenkins-hbase4:43487] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:17,042 INFO [RS:2;jenkins-hbase4:33055] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@75642769{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:17,044 INFO [RS:4;jenkins-hbase4:38101] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@1558c6f4{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:17,044 INFO [RS:5;jenkins-hbase4:43487] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@71820257{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:17,043 INFO [RS:6;jenkins-hbase4:44337] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@5b71a7c8{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:17,047 INFO [RS:5;jenkins-hbase4:43487] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@51056a75{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:17,047 INFO [RS:4;jenkins-hbase4:38101] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@3a1f3aa1{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:17,047 INFO [RS:2;jenkins-hbase4:33055] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:17,047 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:17,047 INFO [RS:2;jenkins-hbase4:33055] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 18:13:17,048 INFO [RS:2;jenkins-hbase4:33055] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 18:13:17,048 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:17,048 INFO [RS:5;jenkins-hbase4:43487] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:17,048 DEBUG [RS:2;jenkins-hbase4:33055] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5db47ee4 to 127.0.0.1:62148 2023-07-15 18:13:17,048 DEBUG [RS:2;jenkins-hbase4:33055] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,048 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:17,048 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,33055,1689444781474; all regions closed. 2023-07-15 18:13:17,048 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,048 INFO [RS:5;jenkins-hbase4:43487] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 18:13:17,048 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 18:13:17,048 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,048 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,049 INFO [RS:5;jenkins-hbase4:43487] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 18:13:17,049 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:17,049 DEBUG [RS:5;jenkins-hbase4:43487] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13929160 to 127.0.0.1:62148 2023-07-15 18:13:17,049 DEBUG [RS:5;jenkins-hbase4:43487] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,049 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43487,1689444794663; all regions closed. 2023-07-15 18:13:17,050 INFO [RS:6;jenkins-hbase4:44337] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@37a18b85{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:17,050 INFO [RS:4;jenkins-hbase4:38101] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:17,050 INFO [RS:4;jenkins-hbase4:38101] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 18:13:17,050 INFO [RS:4;jenkins-hbase4:38101] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 18:13:17,050 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(3305): Received CLOSE for 994da785cf935856bc388ebe43136633 2023-07-15 18:13:17,051 INFO [RS:6;jenkins-hbase4:44337] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 18:13:17,051 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(3305): Received CLOSE for 83c5639bc741d79ce2b4c69e5abc7617 2023-07-15 18:13:17,051 INFO [RS:6;jenkins-hbase4:44337] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 18:13:17,051 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:17,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 994da785cf935856bc388ebe43136633, disabling compactions & flushes 2023-07-15 18:13:17,051 INFO [RS:6;jenkins-hbase4:44337] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 18:13:17,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:17,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:17,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. after waiting 0 ms 2023-07-15 18:13:17,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:17,051 DEBUG [RS:4;jenkins-hbase4:38101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d9cfb61 to 127.0.0.1:62148 2023-07-15 18:13:17,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 994da785cf935856bc388ebe43136633 1/1 column families, dataSize=3.02 KB heapSize=5.57 KB 2023-07-15 18:13:17,051 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:17,053 DEBUG [RS:4;jenkins-hbase4:38101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,053 INFO [RS:4;jenkins-hbase4:38101] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 18:13:17,053 DEBUG [RS:6;jenkins-hbase4:44337] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1422981d to 127.0.0.1:62148 2023-07-15 18:13:17,053 INFO [RS:4;jenkins-hbase4:38101] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 18:13:17,054 INFO [RS:4;jenkins-hbase4:38101] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 18:13:17,054 DEBUG [RS:6;jenkins-hbase4:44337] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,054 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 18:13:17,055 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,44337,1689444796851; all regions closed. 2023-07-15 18:13:17,055 DEBUG [RS:6;jenkins-hbase4:44337] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,055 INFO [RS:6;jenkins-hbase4:44337] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,055 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 18:13:17,055 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1478): Online Regions={994da785cf935856bc388ebe43136633=hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633., 1588230740=hbase:meta,,1.1588230740, 83c5639bc741d79ce2b4c69e5abc7617=hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617.} 2023-07-15 18:13:17,056 DEBUG [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1504): Waiting on 1588230740, 83c5639bc741d79ce2b4c69e5abc7617, 994da785cf935856bc388ebe43136633 2023-07-15 18:13:17,057 INFO [RS:6;jenkins-hbase4:44337] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 18:13:17,058 INFO [RS:6;jenkins-hbase4:44337] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 18:13:17,058 INFO [RS:6;jenkins-hbase4:44337] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 18:13:17,058 INFO [RS:6;jenkins-hbase4:44337] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 18:13:17,057 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 18:13:17,058 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 18:13:17,058 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 18:13:17,058 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 18:13:17,059 INFO [RS:6;jenkins-hbase4:44337] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44337 2023-07-15 18:13:17,058 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 18:13:17,061 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 18:13:17,061 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=6.30 KB heapSize=11.13 KB 2023-07-15 18:13:17,062 DEBUG [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:17,062 INFO [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33055%2C1689444781474.meta:.meta(num 1689444792189) 2023-07-15 18:13:17,063 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:17,063 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,063 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:17,063 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:17,063 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,064 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,064 DEBUG [RS:5;jenkins-hbase4:43487] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:17,064 INFO [RS:5;jenkins-hbase4:43487] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43487%2C1689444794663:(num 1689444794876) 2023-07-15 18:13:17,064 DEBUG [RS:5;jenkins-hbase4:43487] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,064 INFO [RS:5;jenkins-hbase4:43487] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,064 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44337,1689444796851 2023-07-15 18:13:17,067 INFO [RS:5;jenkins-hbase4:43487] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 18:13:17,064 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,068 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44337,1689444796851] 2023-07-15 18:13:17,070 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44337,1689444796851; numProcessing=1 2023-07-15 18:13:17,067 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,070 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 18:13:17,070 INFO [RS:5;jenkins-hbase4:43487] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 18:13:17,071 INFO [RS:5;jenkins-hbase4:43487] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 18:13:17,072 INFO [RS:5;jenkins-hbase4:43487] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 18:13:17,072 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44337,1689444796851 already deleted, retry=false 2023-07-15 18:13:17,073 INFO [RS:5;jenkins-hbase4:43487] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43487 2023-07-15 18:13:17,073 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44337,1689444796851 expired; onlineServers=3 2023-07-15 18:13:17,074 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:17,074 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:17,074 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43487,1689444794663 2023-07-15 18:13:17,074 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,076 DEBUG [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:17,076 INFO [RS:2;jenkins-hbase4:33055] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33055%2C1689444781474:(num 1689444784682) 2023-07-15 18:13:17,076 DEBUG [RS:2;jenkins-hbase4:33055] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,076 INFO [RS:2;jenkins-hbase4:33055] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,076 INFO [RS:2;jenkins-hbase4:33055] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 18:13:17,076 INFO [RS:2;jenkins-hbase4:33055] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 18:13:17,076 INFO [RS:2;jenkins-hbase4:33055] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 18:13:17,076 INFO [RS:2;jenkins-hbase4:33055] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 18:13:17,076 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 18:13:17,077 INFO [RS:2;jenkins-hbase4:33055] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33055 2023-07-15 18:13:17,078 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43487,1689444794663] 2023-07-15 18:13:17,078 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43487,1689444794663; numProcessing=2 2023-07-15 18:13:17,080 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43487,1689444794663 already deleted, retry=false 2023-07-15 18:13:17,080 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43487,1689444794663 expired; onlineServers=2 2023-07-15 18:13:17,080 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:17,080 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,080 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33055,1689444781474 2023-07-15 18:13:17,081 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33055,1689444781474] 2023-07-15 18:13:17,081 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33055,1689444781474; numProcessing=3 2023-07-15 18:13:17,084 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33055,1689444781474 already deleted, retry=false 2023-07-15 18:13:17,084 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33055,1689444781474 expired; onlineServers=1 2023-07-15 18:13:17,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.02 KB at sequenceid=42 (bloomFilter=true), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/96d9d537d8054e94a8392bedae11a580 2023-07-15 18:13:17,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=5.97 KB at sequenceid=70 (bloomFilter=false), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/info/f23bc22508b74bc1999cfa4ba7a4eba7 2023-07-15 18:13:17,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 96d9d537d8054e94a8392bedae11a580 2023-07-15 18:13:17,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/.tmp/m/96d9d537d8054e94a8392bedae11a580 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/96d9d537d8054e94a8392bedae11a580 2023-07-15 18:13:17,096 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f23bc22508b74bc1999cfa4ba7a4eba7 2023-07-15 18:13:17,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 96d9d537d8054e94a8392bedae11a580 2023-07-15 18:13:17,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/96d9d537d8054e94a8392bedae11a580, entries=5, sequenceid=42, filesize=5.3 K 2023-07-15 18:13:17,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.02 KB/3089, heapSize ~5.55 KB/5688, currentSize=0 B/0 for 994da785cf935856bc388ebe43136633 in 48ms, sequenceid=42, compaction requested=false 2023-07-15 18:13:17,103 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 18:13:17,104 DEBUG [StoreCloser-hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f] to archive 2023-07-15 18:13:17,104 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 18:13:17,105 DEBUG [StoreCloser-hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-07-15 18:13:17,108 DEBUG [StoreCloser-hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1 to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/archive/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/1cca1b552a7046428556b030adf432e1 2023-07-15 18:13:17,110 DEBUG [StoreCloser-hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/archive/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/aece4d2030694e0397949a0796daaa0d 2023-07-15 18:13:17,112 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=98 B at sequenceid=70 (bloomFilter=false), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/rep_barrier/c0708924830546f79a35ed0cc37617d9 2023-07-15 18:13:17,112 DEBUG [StoreCloser-hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f to hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/archive/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/m/abdb8e4d3709431ea2ab44d8c4e1c40f 2023-07-15 18:13:17,118 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c0708924830546f79a35ed0cc37617d9 2023-07-15 18:13:17,135 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=70 (bloomFilter=false), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/table/182492caac7749178ea34125139b4a41 2023-07-15 18:13:17,141 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 182492caac7749178ea34125139b4a41 2023-07-15 18:13:17,142 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/info/f23bc22508b74bc1999cfa4ba7a4eba7 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/f23bc22508b74bc1999cfa4ba7a4eba7 2023-07-15 18:13:17,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/rsgroup/994da785cf935856bc388ebe43136633/recovered.edits/45.seqid, newMaxSeqId=45, maxSeqId=28 2023-07-15 18:13:17,151 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f23bc22508b74bc1999cfa4ba7a4eba7 2023-07-15 18:13:17,151 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/info/f23bc22508b74bc1999cfa4ba7a4eba7, entries=22, sequenceid=70, filesize=7.4 K 2023-07-15 18:13:17,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 18:13:17,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:17,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 994da785cf935856bc388ebe43136633: 2023-07-15 18:13:17,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:rsgroup,,1689444785343.994da785cf935856bc388ebe43136633. 2023-07-15 18:13:17,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 83c5639bc741d79ce2b4c69e5abc7617, disabling compactions & flushes 2023-07-15 18:13:17,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:17,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:17,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. after waiting 0 ms 2023-07-15 18:13:17,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:17,154 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/rep_barrier/c0708924830546f79a35ed0cc37617d9 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier/c0708924830546f79a35ed0cc37617d9 2023-07-15 18:13:17,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/namespace/83c5639bc741d79ce2b4c69e5abc7617/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=12 2023-07-15 18:13:17,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:17,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 83c5639bc741d79ce2b4c69e5abc7617: 2023-07-15 18:13:17,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689444785193.83c5639bc741d79ce2b4c69e5abc7617. 2023-07-15 18:13:17,161 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c0708924830546f79a35ed0cc37617d9 2023-07-15 18:13:17,162 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/rep_barrier/c0708924830546f79a35ed0cc37617d9, entries=1, sequenceid=70, filesize=5.0 K 2023-07-15 18:13:17,164 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/.tmp/table/182492caac7749178ea34125139b4a41 as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table/182492caac7749178ea34125139b4a41 2023-07-15 18:13:17,170 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 182492caac7749178ea34125139b4a41 2023-07-15 18:13:17,170 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/table/182492caac7749178ea34125139b4a41, entries=2, sequenceid=70, filesize=5.0 K 2023-07-15 18:13:17,171 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.30 KB/6447, heapSize ~11.09 KB/11352, currentSize=0 B/0 for 1588230740 in 110ms, sequenceid=70, compaction requested=true 2023-07-15 18:13:17,184 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/data/hbase/meta/1588230740/recovered.edits/73.seqid, newMaxSeqId=73, maxSeqId=47 2023-07-15 18:13:17,185 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 18:13:17,185 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 18:13:17,185 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 18:13:17,185 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 18:13:17,220 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:17,220 INFO [RS:2;jenkins-hbase4:33055] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,33055,1689444781474; zookeeper connection closed. 2023-07-15 18:13:17,220 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:33055-0x1016a3001d00003, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:17,220 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6b53bc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6b53bc 2023-07-15 18:13:17,256 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,38101,1689444792984; all regions closed. 2023-07-15 18:13:17,263 DEBUG [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:17,263 INFO [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38101%2C1689444792984.meta:.meta(num 1689444793675) 2023-07-15 18:13:17,268 DEBUG [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/oldWALs 2023-07-15 18:13:17,268 INFO [RS:4;jenkins-hbase4:38101] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38101%2C1689444792984:(num 1689444793236) 2023-07-15 18:13:17,268 DEBUG [RS:4;jenkins-hbase4:38101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,268 INFO [RS:4;jenkins-hbase4:38101] regionserver.LeaseManager(133): Closed leases 2023-07-15 18:13:17,269 INFO [RS:4;jenkins-hbase4:38101] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 18:13:17,269 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 18:13:17,269 INFO [RS:4;jenkins-hbase4:38101] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38101 2023-07-15 18:13:17,271 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38101,1689444792984 2023-07-15 18:13:17,271 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 18:13:17,272 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38101,1689444792984] 2023-07-15 18:13:17,272 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38101,1689444792984; numProcessing=4 2023-07-15 18:13:17,273 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38101,1689444792984 already deleted, retry=false 2023-07-15 18:13:17,273 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38101,1689444792984 expired; onlineServers=0 2023-07-15 18:13:17,273 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42385,1689444778672' ***** 2023-07-15 18:13:17,273 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-15 18:13:17,275 DEBUG [M:0;jenkins-hbase4:42385] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eff15df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 18:13:17,275 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegionServer(1109): Stopping infoServer 2023-07-15 18:13:17,277 INFO [M:0;jenkins-hbase4:42385] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.w.WebAppContext@7c69fb5c{master,/,null,STOPPED}{file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/master} 2023-07-15 18:13:17,277 INFO [M:0;jenkins-hbase4:42385] server.AbstractConnector(383): Stopped ServerConnector@473902f7{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:17,277 INFO [M:0;jenkins-hbase4:42385] session.HouseKeeper(149): node0 Stopped scavenging 2023-07-15 18:13:17,277 INFO [M:0;jenkins-hbase4:42385] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@15103edb{static,/static,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/classes/hbase-webapps/static/,STOPPED} 2023-07-15 18:13:17,277 INFO [M:0;jenkins-hbase4:42385] handler.ContextHandler(1159): Stopped o.a.h.t.o.e.j.s.ServletContextHandler@1740f91f{logs,/logs,file:///home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/hadoop.log.dir/,STOPPED} 2023-07-15 18:13:17,278 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,42385,1689444778672 2023-07-15 18:13:17,278 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-15 18:13:17,278 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42385,1689444778672; all regions closed. 2023-07-15 18:13:17,278 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 18:13:17,278 DEBUG [M:0;jenkins-hbase4:42385] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 18:13:17,278 INFO [M:0;jenkins-hbase4:42385] master.HMaster(1491): Stopping master jetty server 2023-07-15 18:13:17,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 18:13:17,279 INFO [M:0;jenkins-hbase4:42385] server.AbstractConnector(383): Stopped ServerConnector@6a20a799{HTTP/1.1, (http/1.1)}{0.0.0.0:0} 2023-07-15 18:13:17,279 DEBUG [M:0;jenkins-hbase4:42385] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-15 18:13:17,279 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-15 18:13:17,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689444784260] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689444784260,5,FailOnTimeoutGroup] 2023-07-15 18:13:17,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689444784250] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689444784250,5,FailOnTimeoutGroup] 2023-07-15 18:13:17,279 DEBUG [M:0;jenkins-hbase4:42385] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-15 18:13:17,280 INFO [M:0;jenkins-hbase4:42385] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-15 18:13:17,280 INFO [M:0;jenkins-hbase4:42385] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-15 18:13:17,280 INFO [M:0;jenkins-hbase4:42385] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-15 18:13:17,280 DEBUG [M:0;jenkins-hbase4:42385] master.HMaster(1512): Stopping service threads 2023-07-15 18:13:17,280 INFO [M:0;jenkins-hbase4:42385] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-15 18:13:17,280 ERROR [M:0;jenkins-hbase4:42385] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-07-15 18:13:17,281 INFO [M:0;jenkins-hbase4:42385] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-15 18:13:17,281 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-15 18:13:17,281 DEBUG [M:0;jenkins-hbase4:42385] zookeeper.ZKUtil(398): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-15 18:13:17,281 WARN [M:0;jenkins-hbase4:42385] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-15 18:13:17,281 INFO [M:0;jenkins-hbase4:42385] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-15 18:13:17,281 INFO [M:0;jenkins-hbase4:42385] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-15 18:13:17,282 DEBUG [M:0;jenkins-hbase4:42385] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 18:13:17,282 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:17,282 DEBUG [M:0;jenkins-hbase4:42385] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:17,282 DEBUG [M:0;jenkins-hbase4:42385] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 18:13:17,282 DEBUG [M:0;jenkins-hbase4:42385] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:17,282 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=209.34 KB heapSize=252.77 KB 2023-07-15 18:13:17,297 INFO [M:0;jenkins-hbase4:42385] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=209.34 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/734259ae2d4443058aa98bad76147bdc 2023-07-15 18:13:17,303 DEBUG [M:0;jenkins-hbase4:42385] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/734259ae2d4443058aa98bad76147bdc as hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/734259ae2d4443058aa98bad76147bdc 2023-07-15 18:13:17,308 INFO [M:0;jenkins-hbase4:42385] regionserver.HStore(1080): Added hdfs://localhost:39481/user/jenkins/test-data/fe864757-48bf-0aad-d87f-cff875810117/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/734259ae2d4443058aa98bad76147bdc, entries=63, sequenceid=498, filesize=14.0 K 2023-07-15 18:13:17,309 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegion(2948): Finished flush of dataSize ~209.34 KB/214367, heapSize ~252.75 KB/258816, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 27ms, sequenceid=498, compaction requested=false 2023-07-15 18:13:17,311 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 18:13:17,311 DEBUG [M:0;jenkins-hbase4:42385] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 18:13:17,315 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 18:13:17,315 INFO [M:0;jenkins-hbase4:42385] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-15 18:13:17,316 INFO [M:0;jenkins-hbase4:42385] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42385 2023-07-15 18:13:17,317 DEBUG [M:0;jenkins-hbase4:42385] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42385,1689444778672 already deleted, retry=false 2023-07-15 18:13:17,320 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:17,320 INFO [RS:5;jenkins-hbase4:43487] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43487,1689444794663; zookeeper connection closed. 2023-07-15 18:13:17,320 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:43487-0x1016a3001d0000f, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:17,320 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@28d785f1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@28d785f1 2023-07-15 18:13:17,420 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:17,420 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:44337-0x1016a3001d00011, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:17,420 INFO [RS:6;jenkins-hbase4:44337] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,44337,1689444796851; zookeeper connection closed. 2023-07-15 18:13:17,421 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@20921b5c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@20921b5c 2023-07-15 18:13:18,021 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:18,021 INFO [M:0;jenkins-hbase4:42385] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42385,1689444778672; zookeeper connection closed. 2023-07-15 18:13:18,022 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): master:42385-0x1016a3001d00000, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:18,122 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:18,122 INFO [RS:4;jenkins-hbase4:38101] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,38101,1689444792984; zookeeper connection closed. 2023-07-15 18:13:18,122 DEBUG [Listener at localhost/46109-EventThread] zookeeper.ZKWatcher(600): regionserver:38101-0x1016a3001d0000d, quorum=127.0.0.1:62148, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 18:13:18,122 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@58f85093] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@58f85093 2023-07-15 18:13:18,122 INFO [Listener at localhost/46109] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 7 regionserver(s) complete 2023-07-15 18:13:18,123 WARN [Listener at localhost/46109] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 18:13:18,130 INFO [Listener at localhost/46109] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 18:13:18,189 WARN [BP-578561809-172.31.14.131-1689444773408 heartbeating to localhost/127.0.0.1:39481] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-578561809-172.31.14.131-1689444773408 (Datanode Uuid eda77579-bb28-434f-b9ca-03ee57936a73) service to localhost/127.0.0.1:39481 2023-07-15 18:13:18,191 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data5/current/BP-578561809-172.31.14.131-1689444773408] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 18:13:18,191 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data6/current/BP-578561809-172.31.14.131-1689444773408] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 18:13:18,236 WARN [Listener at localhost/46109] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 18:13:18,334 INFO [Listener at localhost/46109] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 18:13:18,438 WARN [BP-578561809-172.31.14.131-1689444773408 heartbeating to localhost/127.0.0.1:39481] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 18:13:18,438 WARN [BP-578561809-172.31.14.131-1689444773408 heartbeating to localhost/127.0.0.1:39481] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-578561809-172.31.14.131-1689444773408 (Datanode Uuid 8050b55f-8f12-47d7-a080-9ddfbceb52d3) service to localhost/127.0.0.1:39481 2023-07-15 18:13:18,438 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data3/current/BP-578561809-172.31.14.131-1689444773408] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 18:13:18,439 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data4/current/BP-578561809-172.31.14.131-1689444773408] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 18:13:18,440 WARN [Listener at localhost/46109] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 18:13:18,550 INFO [Listener at localhost/46109] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 18:13:18,655 WARN [BP-578561809-172.31.14.131-1689444773408 heartbeating to localhost/127.0.0.1:39481] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 18:13:18,655 WARN [BP-578561809-172.31.14.131-1689444773408 heartbeating to localhost/127.0.0.1:39481] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-578561809-172.31.14.131-1689444773408 (Datanode Uuid 83b629db-5560-495e-b685-389dfc1aa428) service to localhost/127.0.0.1:39481 2023-07-15 18:13:18,655 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data1/current/BP-578561809-172.31.14.131-1689444773408] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 18:13:18,656 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-rsgroup/target/test-data/2a8ca982-34a2-8b3a-733a-e835b1cb90a0/cluster_07a76730-aa44-a66c-67ce-8d3ef2cea80a/dfs/data/data2/current/BP-578561809-172.31.14.131-1689444773408] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 18:13:18,685 INFO [Listener at localhost/46109] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 18:13:18,803 INFO [Listener at localhost/46109] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-15 18:13:18,852 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.rsgroup.TestRSGroupsBase$CPMasterObserver 2023-07-15 18:13:18,852 INFO [Listener at localhost/46109] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-15 18:13:18,852 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint