2023-02-11 04:00:48,969 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929 2023-02-11 04:00:48,982 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2 timeout: 13 mins 2023-02-11 04:00:49,020 INFO [Time-limited test] hbase.ResourceChecker(147): before: client.TestAsyncClusterAdminApi2#testStop Thread=8, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=229, ProcessCount=169, AvailableMemoryMB=3723 2023-02-11 04:00:49,026 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 04:00:49,027 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01, deleteOnExit=true 2023-02-11 04:00:49,027 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 04:00:49,028 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/test.cache.data in system properties and HBase conf 2023-02-11 04:00:49,028 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 04:00:49,028 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/hadoop.log.dir in system properties and HBase conf 2023-02-11 04:00:49,029 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 04:00:49,029 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 04:00:49,030 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 04:00:49,134 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-11 04:00:49,518 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 04:00:49,523 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 04:00:49,524 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 04:00:49,524 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 04:00:49,525 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 04:00:49,525 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 04:00:49,526 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 04:00:49,526 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 04:00:49,527 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 04:00:49,527 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 04:00:49,528 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/nfs.dump.dir in system properties and HBase conf 2023-02-11 04:00:49,528 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/java.io.tmpdir in system properties and HBase conf 2023-02-11 04:00:49,528 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 04:00:49,529 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 04:00:49,529 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 04:00:50,004 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 04:00:50,008 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 04:00:50,290 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-11 04:00:50,440 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-11 04:00:50,454 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:50,490 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:50,521 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/java.io.tmpdir/Jetty_localhost_37241_hdfs____x1wa2d/webapp 2023-02-11 04:00:50,674 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37241 2023-02-11 04:00:50,685 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 04:00:50,686 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 04:00:51,092 WARN [Listener at localhost/42141] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:51,155 WARN [Listener at localhost/42141] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:51,172 WARN [Listener at localhost/42141] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:51,178 INFO [Listener at localhost/42141] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:51,182 INFO [Listener at localhost/42141] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/java.io.tmpdir/Jetty_localhost_36095_datanode____.hcma34/webapp 2023-02-11 04:00:51,277 INFO [Listener at localhost/42141] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36095 2023-02-11 04:00:51,577 WARN [Listener at localhost/38665] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:51,588 WARN [Listener at localhost/38665] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:51,592 WARN [Listener at localhost/38665] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:51,594 INFO [Listener at localhost/38665] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:51,601 INFO [Listener at localhost/38665] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/java.io.tmpdir/Jetty_localhost_34983_datanode____swcd3u/webapp 2023-02-11 04:00:51,727 INFO [Listener at localhost/38665] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34983 2023-02-11 04:00:51,743 WARN [Listener at localhost/36895] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:51,757 WARN [Listener at localhost/36895] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:51,759 WARN [Listener at localhost/36895] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:51,761 INFO [Listener at localhost/36895] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:51,768 INFO [Listener at localhost/36895] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/java.io.tmpdir/Jetty_localhost_45383_datanode____zazoac/webapp 2023-02-11 04:00:51,881 INFO [Listener at localhost/36895] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45383 2023-02-11 04:00:51,889 WARN [Listener at localhost/41065] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:52,059 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6c22ac67cb9daea5: Processing first storage report for DS-182f3421-ff21-4527-8c73-15a0f94a4bae from datanode 84c323d8-9611-48a9-996c-60ff152a6d06 2023-02-11 04:00:52,060 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6c22ac67cb9daea5: from storage DS-182f3421-ff21-4527-8c73-15a0f94a4bae node DatanodeRegistration(127.0.0.1:40255, datanodeUuid=84c323d8-9611-48a9-996c-60ff152a6d06, infoPort=36931, infoSecurePort=0, ipcPort=41065, storageInfo=lv=-57;cid=testClusterID;nsid=1962379674;c=1676088050078), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbd464917e42aa03e: Processing first storage report for DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87 from datanode 93c9d8c5-f8b2-4201-b019-dba9d527bdb7 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbd464917e42aa03e: from storage DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87 node DatanodeRegistration(127.0.0.1:33799, datanodeUuid=93c9d8c5-f8b2-4201-b019-dba9d527bdb7, infoPort=42159, infoSecurePort=0, ipcPort=38665, storageInfo=lv=-57;cid=testClusterID;nsid=1962379674;c=1676088050078), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb51dc06731df3b20: Processing first storage report for DS-736eb529-4df5-4254-8a0d-c062c101bf4c from datanode 5dcf6bcd-f24d-4994-8ecb-5ee6400ae0c8 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb51dc06731df3b20: from storage DS-736eb529-4df5-4254-8a0d-c062c101bf4c node DatanodeRegistration(127.0.0.1:45305, datanodeUuid=5dcf6bcd-f24d-4994-8ecb-5ee6400ae0c8, infoPort=38553, infoSecurePort=0, ipcPort=36895, storageInfo=lv=-57;cid=testClusterID;nsid=1962379674;c=1676088050078), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6c22ac67cb9daea5: Processing first storage report for DS-39c5ecd8-f437-403f-ad4a-2465aa72e385 from datanode 84c323d8-9611-48a9-996c-60ff152a6d06 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6c22ac67cb9daea5: from storage DS-39c5ecd8-f437-403f-ad4a-2465aa72e385 node DatanodeRegistration(127.0.0.1:40255, datanodeUuid=84c323d8-9611-48a9-996c-60ff152a6d06, infoPort=36931, infoSecurePort=0, ipcPort=41065, storageInfo=lv=-57;cid=testClusterID;nsid=1962379674;c=1676088050078), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:52,061 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbd464917e42aa03e: Processing first storage report for DS-137e7196-702d-4134-991b-edad51eae5eb from datanode 93c9d8c5-f8b2-4201-b019-dba9d527bdb7 2023-02-11 04:00:52,062 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbd464917e42aa03e: from storage DS-137e7196-702d-4134-991b-edad51eae5eb node DatanodeRegistration(127.0.0.1:33799, datanodeUuid=93c9d8c5-f8b2-4201-b019-dba9d527bdb7, infoPort=42159, infoSecurePort=0, ipcPort=38665, storageInfo=lv=-57;cid=testClusterID;nsid=1962379674;c=1676088050078), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:52,062 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb51dc06731df3b20: Processing first storage report for DS-e3ecc836-6e1f-48c3-bce2-edee4b6a5694 from datanode 5dcf6bcd-f24d-4994-8ecb-5ee6400ae0c8 2023-02-11 04:00:52,062 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb51dc06731df3b20: from storage DS-e3ecc836-6e1f-48c3-bce2-edee4b6a5694 node DatanodeRegistration(127.0.0.1:45305, datanodeUuid=5dcf6bcd-f24d-4994-8ecb-5ee6400ae0c8, infoPort=38553, infoSecurePort=0, ipcPort=36895, storageInfo=lv=-57;cid=testClusterID;nsid=1962379674;c=1676088050078), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:52,271 DEBUG [Listener at localhost/41065] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929 2023-02-11 04:00:52,332 INFO [Listener at localhost/41065] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/zookeeper_0, clientPort=53335, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 04:00:52,345 INFO [Listener at localhost/41065] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53335 2023-02-11 04:00:52,355 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:52,357 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:53,027 INFO [Listener at localhost/41065] util.FSUtils(479): Created version file at hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8 with version=8 2023-02-11 04:00:53,027 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/hbase-staging 2023-02-11 04:00:53,339 INFO [Listener at localhost/41065] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 04:00:53,786 INFO [Listener at localhost/41065] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:53,817 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:53,817 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:53,817 INFO [Listener at localhost/41065] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:53,817 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:53,818 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:53,956 INFO [Listener at localhost/41065] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:54,027 DEBUG [Listener at localhost/41065] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-11 04:00:54,119 INFO [Listener at localhost/41065] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36021 2023-02-11 04:00:54,129 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,131 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,150 INFO [Listener at localhost/41065] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36021 connecting to ZooKeeper ensemble=127.0.0.1:53335 2023-02-11 04:00:54,192 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:360210x0, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:54,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36021-0x1004189f4230000 connected 2023-02-11 04:00:54,220 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:54,221 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:54,224 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:54,232 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36021 2023-02-11 04:00:54,232 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36021 2023-02-11 04:00:54,232 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36021 2023-02-11 04:00:54,233 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36021 2023-02-11 04:00:54,233 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36021 2023-02-11 04:00:54,239 INFO [Listener at localhost/41065] master.HMaster(439): hbase.rootdir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8, hbase.cluster.distributed=false 2023-02-11 04:00:54,304 INFO [Listener at localhost/41065] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:54,305 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,305 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,305 INFO [Listener at localhost/41065] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:54,305 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,305 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:54,310 INFO [Listener at localhost/41065] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:54,313 INFO [Listener at localhost/41065] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36649 2023-02-11 04:00:54,315 INFO [Listener at localhost/41065] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:54,321 DEBUG [Listener at localhost/41065] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 04:00:54,323 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,325 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,326 INFO [Listener at localhost/41065] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36649 connecting to ZooKeeper ensemble=127.0.0.1:53335 2023-02-11 04:00:54,330 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:366490x0, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:54,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36649-0x1004189f4230001 connected 2023-02-11 04:00:54,331 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:54,333 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:54,333 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:54,334 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36649 2023-02-11 04:00:54,334 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36649 2023-02-11 04:00:54,335 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36649 2023-02-11 04:00:54,335 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36649 2023-02-11 04:00:54,335 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36649 2023-02-11 04:00:54,347 INFO [Listener at localhost/41065] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:54,347 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,347 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,348 INFO [Listener at localhost/41065] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:54,348 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,348 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:54,349 INFO [Listener at localhost/41065] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:54,350 INFO [Listener at localhost/41065] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33927 2023-02-11 04:00:54,350 INFO [Listener at localhost/41065] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:54,351 DEBUG [Listener at localhost/41065] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 04:00:54,352 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,354 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,356 INFO [Listener at localhost/41065] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33927 connecting to ZooKeeper ensemble=127.0.0.1:53335 2023-02-11 04:00:54,360 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:339270x0, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:54,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33927-0x1004189f4230002 connected 2023-02-11 04:00:54,361 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:54,362 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:54,363 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:54,363 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33927 2023-02-11 04:00:54,364 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33927 2023-02-11 04:00:54,364 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33927 2023-02-11 04:00:54,365 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33927 2023-02-11 04:00:54,365 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33927 2023-02-11 04:00:54,377 INFO [Listener at localhost/41065] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:54,378 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,378 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,378 INFO [Listener at localhost/41065] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:54,378 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:54,378 INFO [Listener at localhost/41065] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:54,378 INFO [Listener at localhost/41065] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:54,380 INFO [Listener at localhost/41065] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43901 2023-02-11 04:00:54,380 INFO [Listener at localhost/41065] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:54,381 DEBUG [Listener at localhost/41065] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 04:00:54,382 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,384 INFO [Listener at localhost/41065] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,385 INFO [Listener at localhost/41065] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43901 connecting to ZooKeeper ensemble=127.0.0.1:53335 2023-02-11 04:00:54,389 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:439010x0, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:54,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43901-0x1004189f4230003 connected 2023-02-11 04:00:54,390 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:54,390 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:54,391 DEBUG [Listener at localhost/41065] zookeeper.ZKUtil(164): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:54,392 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43901 2023-02-11 04:00:54,392 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43901 2023-02-11 04:00:54,392 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43901 2023-02-11 04:00:54,393 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43901 2023-02-11 04:00:54,393 DEBUG [Listener at localhost/41065] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43901 2023-02-11 04:00:54,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:54,403 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 04:00:54,404 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:54,421 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:54,421 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:54,421 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:54,421 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:54,421 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:54,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 04:00:54,424 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,36021,1676088053177 from backup master directory 2023-02-11 04:00:54,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 04:00:54,428 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:54,428 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 04:00:54,429 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:54,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:54,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-11 04:00:54,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-11 04:00:54,520 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/hbase.id with ID: c7b1d5f3-a73c-46ee-9b41-a117677cdadd 2023-02-11 04:00:54,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:54,579 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:54,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7d72e187 to 127.0.0.1:53335 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:54,659 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61a61312, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:54,681 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:54,683 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 04:00:54,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-11 04:00:54,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-11 04:00:54,701 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:54,705 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:54,706 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:54,737 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store-tmp 2023-02-11 04:00:54,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:54,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 04:00:54,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:54,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:54,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 04:00:54,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:54,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:54,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676088054799Disabling compacts and flushes for region at 1676088054800 (+1 ms)Disabling writes for close at 1676088054800Writing region close event to WAL at 1676088054800Closed at 1676088054800 2023-02-11 04:00:54,802 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/WALs/jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:54,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36021%2C1676088053177, suffix=, logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/WALs/jenkins-hbase4.apache.org,36021,1676088053177, archiveDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/oldWALs, maxLogs=10 2023-02-11 04:00:54,872 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK] 2023-02-11 04:00:54,872 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK] 2023-02-11 04:00:54,872 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK] 2023-02-11 04:00:54,880 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:54,946 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/WALs/jenkins-hbase4.apache.org,36021,1676088053177/jenkins-hbase4.apache.org%2C36021%2C1676088053177.1676088054830 2023-02-11 04:00:54,947 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK], DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK], DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK]] 2023-02-11 04:00:54,947 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:54,948 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:54,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:54,952 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:55,011 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:55,017 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 04:00:55,041 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 04:00:55,052 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:55,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:55,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:55,072 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:55,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:55,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62542787, jitterRate=-0.06803984940052032}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:55,077 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676088054976Initializing all the Stores at 1676088054978 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088054979 (+1 ms)Cleaning up temporary data from old regions at 1676088055065 (+86 ms)Cleaning up detritus from prior splits at 1676088055066 (+1 ms)Region opened successfully at 1676088055077 (+11 ms) 2023-02-11 04:00:55,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 04:00:55,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 04:00:55,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 04:00:55,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 04:00:55,101 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-11 04:00:55,133 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-11 04:00:55,133 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 04:00:55,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 04:00:55,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 04:00:55,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 04:00:55,188 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 04:00:55,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 04:00:55,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 04:00:55,198 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 04:00:55,201 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:55,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 04:00:55,203 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 04:00:55,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 04:00:55,219 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:55,219 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:55,219 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:55,219 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:55,219 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:55,220 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,36021,1676088053177, sessionid=0x1004189f4230000, setting cluster-up flag (Was=false) 2023-02-11 04:00:55,236 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:55,242 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 04:00:55,243 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:55,247 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:55,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 04:00:55,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:55,257 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/.hbase-snapshot/.tmp 2023-02-11 04:00:55,297 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(952): ClusterId : c7b1d5f3-a73c-46ee-9b41-a117677cdadd 2023-02-11 04:00:55,298 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(952): ClusterId : c7b1d5f3-a73c-46ee-9b41-a117677cdadd 2023-02-11 04:00:55,297 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(952): ClusterId : c7b1d5f3-a73c-46ee-9b41-a117677cdadd 2023-02-11 04:00:55,303 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:55,303 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:55,303 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:55,310 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:55,310 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:55,310 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:55,310 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:55,310 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:55,310 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:55,315 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:55,315 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:55,315 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:55,317 DEBUG [RS:0;jenkins-hbase4:36649] zookeeper.ReadOnlyZKClient(139): Connect 0x422bdf92 to 127.0.0.1:53335 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:55,318 DEBUG [RS:2;jenkins-hbase4:43901] zookeeper.ReadOnlyZKClient(139): Connect 0x6f39620d to 127.0.0.1:53335 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:55,318 DEBUG [RS:1;jenkins-hbase4:33927] zookeeper.ReadOnlyZKClient(139): Connect 0x49f65a87 to 127.0.0.1:53335 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:55,331 DEBUG [RS:0;jenkins-hbase4:36649] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cec29a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:55,332 DEBUG [RS:2;jenkins-hbase4:43901] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61fe919a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:55,332 DEBUG [RS:1;jenkins-hbase4:33927] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4731adc9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:55,332 DEBUG [RS:0;jenkins-hbase4:36649] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48d3b4e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:55,332 DEBUG [RS:2;jenkins-hbase4:43901] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65c1097d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:55,332 DEBUG [RS:1;jenkins-hbase4:33927] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@430e31f7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:55,358 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:43901 2023-02-11 04:00:55,361 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:36649 2023-02-11 04:00:55,361 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:33927 2023-02-11 04:00:55,363 INFO [RS:0;jenkins-hbase4:36649] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:55,363 INFO [RS:1;jenkins-hbase4:33927] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:55,363 INFO [RS:1;jenkins-hbase4:33927] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:55,363 INFO [RS:2;jenkins-hbase4:43901] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:55,363 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:55,363 INFO [RS:0;jenkins-hbase4:36649] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:55,363 INFO [RS:2;jenkins-hbase4:43901] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:55,363 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:55,364 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:55,366 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36021,1676088053177 with isa=jenkins-hbase4.apache.org/172.31.14.131:36649, startcode=1676088054304 2023-02-11 04:00:55,366 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36021,1676088053177 with isa=jenkins-hbase4.apache.org/172.31.14.131:43901, startcode=1676088054377 2023-02-11 04:00:55,366 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36021,1676088053177 with isa=jenkins-hbase4.apache.org/172.31.14.131:33927, startcode=1676088054347 2023-02-11 04:00:55,385 DEBUG [RS:1;jenkins-hbase4:33927] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:55,385 DEBUG [RS:2;jenkins-hbase4:43901] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:55,385 DEBUG [RS:0;jenkins-hbase4:36649] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:55,389 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:55,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,401 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676088085401 2023-02-11 04:00:55,404 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 04:00:55,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 04:00:55,424 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:55,425 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 04:00:55,426 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 04:00:55,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 04:00:55,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 04:00:55,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 04:00:55,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 04:00:55,430 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:55,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 04:00:55,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 04:00:55,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 04:00:55,434 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 04:00:55,440 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088055436,5,FailOnTimeoutGroup] 2023-02-11 04:00:55,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088055440,5,FailOnTimeoutGroup] 2023-02-11 04:00:55,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 04:00:55,445 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56671, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:55,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,445 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35035, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:55,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,445 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41517, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:55,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36021] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,462 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36021] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,463 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36021] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,482 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:55,483 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:55,483 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8 2023-02-11 04:00:55,485 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8 2023-02-11 04:00:55,485 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8 2023-02-11 04:00:55,485 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8 2023-02-11 04:00:55,486 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42141 2023-02-11 04:00:55,485 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42141 2023-02-11 04:00:55,486 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:55,486 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42141 2023-02-11 04:00:55,486 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:55,486 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:55,496 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:55,497 DEBUG [RS:2;jenkins-hbase4:43901] zookeeper.ZKUtil(162): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,497 DEBUG [RS:0;jenkins-hbase4:36649] zookeeper.ZKUtil(162): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,497 DEBUG [RS:1;jenkins-hbase4:33927] zookeeper.ZKUtil(162): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,497 WARN [RS:2;jenkins-hbase4:43901] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:55,498 WARN [RS:1;jenkins-hbase4:33927] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:55,497 WARN [RS:0;jenkins-hbase4:36649] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:55,498 INFO [RS:1;jenkins-hbase4:33927] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:55,498 INFO [RS:2;jenkins-hbase4:43901] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:55,498 INFO [RS:0;jenkins-hbase4:36649] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:55,499 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,499 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,499 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,499 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36649,1676088054304] 2023-02-11 04:00:55,499 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43901,1676088054377] 2023-02-11 04:00:55,499 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33927,1676088054347] 2023-02-11 04:00:55,523 DEBUG [RS:1;jenkins-hbase4:33927] zookeeper.ZKUtil(162): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,524 DEBUG [RS:0;jenkins-hbase4:36649] zookeeper.ZKUtil(162): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,523 DEBUG [RS:2;jenkins-hbase4:43901] zookeeper.ZKUtil(162): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,524 DEBUG [RS:1;jenkins-hbase4:33927] zookeeper.ZKUtil(162): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,524 DEBUG [RS:0;jenkins-hbase4:36649] zookeeper.ZKUtil(162): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,524 DEBUG [RS:2;jenkins-hbase4:43901] zookeeper.ZKUtil(162): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,525 DEBUG [RS:1;jenkins-hbase4:33927] zookeeper.ZKUtil(162): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,525 DEBUG [RS:0;jenkins-hbase4:36649] zookeeper.ZKUtil(162): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,525 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:55,525 DEBUG [RS:2;jenkins-hbase4:43901] zookeeper.ZKUtil(162): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,529 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 04:00:55,531 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/info 2023-02-11 04:00:55,532 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 04:00:55,533 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:55,533 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 04:00:55,535 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:55,535 DEBUG [RS:0;jenkins-hbase4:36649] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:55,536 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:55,536 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:55,537 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 04:00:55,538 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:55,538 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 04:00:55,540 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/table 2023-02-11 04:00:55,542 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 04:00:55,543 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:55,544 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740 2023-02-11 04:00:55,546 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740 2023-02-11 04:00:55,548 INFO [RS:2;jenkins-hbase4:43901] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:55,548 INFO [RS:1;jenkins-hbase4:33927] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:55,549 INFO [RS:0;jenkins-hbase4:36649] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:55,551 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 04:00:55,554 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 04:00:55,565 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:55,566 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74679540, jitterRate=0.1128118634223938}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 04:00:55,566 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676088055525Initializing all the Stores at 1676088055528 (+3 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088055528Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088055528Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088055528Cleaning up temporary data from old regions at 1676088055548 (+20 ms)Cleaning up detritus from prior splits at 1676088055549 (+1 ms)Region opened successfully at 1676088055566 (+17 ms) 2023-02-11 04:00:55,566 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 04:00:55,566 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 04:00:55,567 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 04:00:55,567 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 04:00:55,567 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 04:00:55,568 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:55,568 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676088055566Disabling compacts and flushes for region at 1676088055566Disabling writes for close at 1676088055567 (+1 ms)Writing region close event to WAL at 1676088055568 (+1 ms)Closed at 1676088055568 2023-02-11 04:00:55,574 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:55,574 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 04:00:55,578 INFO [RS:1;jenkins-hbase4:33927] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:55,578 INFO [RS:0;jenkins-hbase4:36649] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:55,578 INFO [RS:2;jenkins-hbase4:43901] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:55,583 INFO [RS:1;jenkins-hbase4:33927] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:55,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 04:00:55,583 INFO [RS:2;jenkins-hbase4:43901] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:55,583 INFO [RS:0;jenkins-hbase4:36649] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:55,584 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,584 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,584 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,585 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:55,588 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:55,588 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:55,596 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,596 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,596 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,596 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,596 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:55,597 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,597 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,598 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,598 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,598 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 04:00:55,598 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,598 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,598 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,598 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:55,599 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:55,599 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:1;jenkins-hbase4:33927] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,600 DEBUG [RS:2;jenkins-hbase4:43901] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,599 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,600 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,600 DEBUG [RS:0;jenkins-hbase4:36649] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:55,601 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:55,602 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,602 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,618 INFO [RS:1;jenkins-hbase4:33927] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:55,619 INFO [RS:2;jenkins-hbase4:43901] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:55,620 INFO [RS:0;jenkins-hbase4:36649] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:55,621 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43901,1676088054377-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,621 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36649,1676088054304-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,621 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33927,1676088054347-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:55,639 INFO [RS:0;jenkins-hbase4:36649] regionserver.Replication(203): jenkins-hbase4.apache.org,36649,1676088054304 started 2023-02-11 04:00:55,639 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36649,1676088054304, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36649, sessionid=0x1004189f4230001 2023-02-11 04:00:55,639 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:00:55,639 DEBUG [RS:0;jenkins-hbase4:36649] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,639 DEBUG [RS:0;jenkins-hbase4:36649] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36649,1676088054304' 2023-02-11 04:00:55,639 DEBUG [RS:0;jenkins-hbase4:36649] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:00:55,640 DEBUG [RS:0;jenkins-hbase4:36649] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:00:55,640 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:00:55,640 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:00:55,640 DEBUG [RS:0;jenkins-hbase4:36649] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:55,640 DEBUG [RS:0;jenkins-hbase4:36649] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36649,1676088054304' 2023-02-11 04:00:55,641 DEBUG [RS:0;jenkins-hbase4:36649] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:00:55,641 DEBUG [RS:0;jenkins-hbase4:36649] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:00:55,641 DEBUG [RS:0;jenkins-hbase4:36649] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:00:55,642 INFO [RS:0;jenkins-hbase4:36649] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:00:55,642 INFO [RS:0;jenkins-hbase4:36649] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:00:55,642 INFO [RS:2;jenkins-hbase4:43901] regionserver.Replication(203): jenkins-hbase4.apache.org,43901,1676088054377 started 2023-02-11 04:00:55,642 INFO [RS:1;jenkins-hbase4:33927] regionserver.Replication(203): jenkins-hbase4.apache.org,33927,1676088054347 started 2023-02-11 04:00:55,642 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43901,1676088054377, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43901, sessionid=0x1004189f4230003 2023-02-11 04:00:55,642 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33927,1676088054347, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33927, sessionid=0x1004189f4230002 2023-02-11 04:00:55,642 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:00:55,642 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:00:55,642 DEBUG [RS:2;jenkins-hbase4:43901] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,643 DEBUG [RS:1;jenkins-hbase4:33927] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,643 DEBUG [RS:1;jenkins-hbase4:33927] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33927,1676088054347' 2023-02-11 04:00:55,643 DEBUG [RS:1;jenkins-hbase4:33927] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:00:55,643 DEBUG [RS:2;jenkins-hbase4:43901] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43901,1676088054377' 2023-02-11 04:00:55,643 DEBUG [RS:2;jenkins-hbase4:43901] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:00:55,644 DEBUG [RS:1;jenkins-hbase4:33927] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:00:55,644 DEBUG [RS:2;jenkins-hbase4:43901] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:00:55,644 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:00:55,644 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:00:55,644 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:00:55,644 DEBUG [RS:1;jenkins-hbase4:33927] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:55,645 DEBUG [RS:1;jenkins-hbase4:33927] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33927,1676088054347' 2023-02-11 04:00:55,645 DEBUG [RS:1;jenkins-hbase4:33927] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:00:55,645 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:00:55,645 DEBUG [RS:2;jenkins-hbase4:43901] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,645 DEBUG [RS:2;jenkins-hbase4:43901] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43901,1676088054377' 2023-02-11 04:00:55,645 DEBUG [RS:2;jenkins-hbase4:43901] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:00:55,645 DEBUG [RS:1;jenkins-hbase4:33927] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:00:55,646 DEBUG [RS:2;jenkins-hbase4:43901] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:00:55,646 DEBUG [RS:1;jenkins-hbase4:33927] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:00:55,646 INFO [RS:1;jenkins-hbase4:33927] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:00:55,646 INFO [RS:1;jenkins-hbase4:33927] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:00:55,647 DEBUG [RS:2;jenkins-hbase4:43901] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:00:55,647 INFO [RS:2;jenkins-hbase4:43901] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:00:55,647 INFO [RS:2;jenkins-hbase4:43901] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:00:55,752 INFO [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43901%2C1676088054377, suffix=, logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,43901,1676088054377, archiveDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs, maxLogs=32 2023-02-11 04:00:55,753 INFO [RS:0;jenkins-hbase4:36649] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36649%2C1676088054304, suffix=, logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,36649,1676088054304, archiveDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs, maxLogs=32 2023-02-11 04:00:55,753 INFO [RS:1;jenkins-hbase4:33927] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33927%2C1676088054347, suffix=, logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,33927,1676088054347, archiveDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs, maxLogs=32 2023-02-11 04:00:55,754 DEBUG [jenkins-hbase4:36021] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 04:00:55,758 DEBUG [jenkins-hbase4:36021] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:55,764 DEBUG [jenkins-hbase4:36021] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:55,764 DEBUG [jenkins-hbase4:36021] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:55,764 DEBUG [jenkins-hbase4:36021] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:55,764 DEBUG [jenkins-hbase4:36021] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:55,767 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43901,1676088054377, state=OPENING 2023-02-11 04:00:55,783 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 04:00:55,785 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:55,785 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK] 2023-02-11 04:00:55,787 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 04:00:55,794 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK] 2023-02-11 04:00:55,795 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK] 2023-02-11 04:00:55,796 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK] 2023-02-11 04:00:55,796 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK] 2023-02-11 04:00:55,796 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK] 2023-02-11 04:00:55,796 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43901,1676088054377}] 2023-02-11 04:00:55,800 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK] 2023-02-11 04:00:55,800 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK] 2023-02-11 04:00:55,806 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK] 2023-02-11 04:00:55,817 INFO [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,43901,1676088054377/jenkins-hbase4.apache.org%2C43901%2C1676088054377.1676088055760 2023-02-11 04:00:55,817 INFO [RS:1;jenkins-hbase4:33927] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,33927,1676088054347/jenkins-hbase4.apache.org%2C33927%2C1676088054347.1676088055760 2023-02-11 04:00:55,817 INFO [RS:0;jenkins-hbase4:36649] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,36649,1676088054304/jenkins-hbase4.apache.org%2C36649%2C1676088054304.1676088055760 2023-02-11 04:00:55,818 DEBUG [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK], DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK], DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK]] 2023-02-11 04:00:55,818 DEBUG [RS:0;jenkins-hbase4:36649] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK], DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK], DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK]] 2023-02-11 04:00:55,820 DEBUG [RS:1;jenkins-hbase4:33927] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK], DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK], DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK]] 2023-02-11 04:00:55,994 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:55,996 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:00:55,999 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:00:56,012 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 04:00:56,012 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:56,015 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43901%2C1676088054377.meta, suffix=.meta, logDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,43901,1676088054377, archiveDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs, maxLogs=32 2023-02-11 04:00:56,037 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK] 2023-02-11 04:00:56,038 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK] 2023-02-11 04:00:56,040 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK] 2023-02-11 04:00:56,045 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/WALs/jenkins-hbase4.apache.org,43901,1676088054377/jenkins-hbase4.apache.org%2C43901%2C1676088054377.meta.1676088056018.meta 2023-02-11 04:00:56,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33799,DS-08bb417d-c2ec-4621-b9f3-4ed382ac0b87,DISK], DatanodeInfoWithStorage[127.0.0.1:45305,DS-736eb529-4df5-4254-8a0d-c062c101bf4c,DISK], DatanodeInfoWithStorage[127.0.0.1:40255,DS-182f3421-ff21-4527-8c73-15a0f94a4bae,DISK]] 2023-02-11 04:00:56,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:56,051 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 04:00:56,068 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 04:00:56,073 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 04:00:56,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 04:00:56,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:56,079 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 04:00:56,079 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 04:00:56,081 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 04:00:56,083 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/info 2023-02-11 04:00:56,083 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/info 2023-02-11 04:00:56,084 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 04:00:56,084 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:56,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 04:00:56,086 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:56,086 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:56,087 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 04:00:56,087 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:56,088 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 04:00:56,089 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/table 2023-02-11 04:00:56,089 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/table 2023-02-11 04:00:56,089 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 04:00:56,090 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:56,092 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740 2023-02-11 04:00:56,095 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740 2023-02-11 04:00:56,098 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 04:00:56,101 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 04:00:56,102 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62696053, jitterRate=-0.06575600802898407}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 04:00:56,102 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676088056079Writing region info on filesystem at 1676088056079Initializing all the Stores at 1676088056081 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088056081Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088056081Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088056081Cleaning up temporary data from old regions at 1676088056097 (+16 ms)Cleaning up detritus from prior splits at 1676088056097Running coprocessor post-open hooks at 1676088056102 (+5 ms)Region opened successfully at 1676088056102 2023-02-11 04:00:56,114 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676088055986 2023-02-11 04:00:56,131 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 04:00:56,132 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 04:00:56,133 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43901,1676088054377, state=OPEN 2023-02-11 04:00:56,136 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 04:00:56,136 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 04:00:56,140 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 04:00:56,140 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43901,1676088054377 in 340 msec 2023-02-11 04:00:56,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 04:00:56,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 557 msec 2023-02-11 04:00:56,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 848 msec 2023-02-11 04:00:56,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676088054418, completionTime=-1 2023-02-11 04:00:56,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 04:00:56,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 04:00:56,218 DEBUG [hconnection-0x539f6431-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:56,221 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44790, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:56,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 04:00:56,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676088116237 2023-02-11 04:00:56,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676088176237 2023-02-11 04:00:56,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 84 msec 2023-02-11 04:00:56,266 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36021,1676088053177-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:56,266 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36021,1676088053177-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:56,266 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36021,1676088053177-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:56,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:36021, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:56,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:56,275 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 04:00:56,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 04:00:56,285 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:56,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 04:00:56,297 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 04:00:56,299 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 04:00:56,321 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/.tmp/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,323 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/.tmp/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80 empty. 2023-02-11 04:00:56,324 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/.tmp/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,324 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 04:00:56,363 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:56,365 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => c2bcbbf42dd249487e9caccf35ab2b80, NAME => 'hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/.tmp 2023-02-11 04:00:56,386 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:56,386 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing c2bcbbf42dd249487e9caccf35ab2b80, disabling compactions & flushes 2023-02-11 04:00:56,386 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,386 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,386 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. after waiting 0 ms 2023-02-11 04:00:56,386 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,386 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,386 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for c2bcbbf42dd249487e9caccf35ab2b80: Waiting for close lock at 1676088056386Disabling compacts and flushes for region at 1676088056386Disabling writes for close at 1676088056386Writing region close event to WAL at 1676088056386Closed at 1676088056386 2023-02-11 04:00:56,390 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 04:00:56,407 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088056393"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088056393"}]},"ts":"1676088056393"} 2023-02-11 04:00:56,433 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 04:00:56,435 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 04:00:56,440 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088056436"}]},"ts":"1676088056436"} 2023-02-11 04:00:56,444 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 04:00:56,448 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:56,449 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:56,449 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:56,449 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:56,449 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:56,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c2bcbbf42dd249487e9caccf35ab2b80, ASSIGN}] 2023-02-11 04:00:56,454 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c2bcbbf42dd249487e9caccf35ab2b80, ASSIGN 2023-02-11 04:00:56,456 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c2bcbbf42dd249487e9caccf35ab2b80, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33927,1676088054347; forceNewPlan=false, retain=false 2023-02-11 04:00:56,608 INFO [jenkins-hbase4:36021] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 04:00:56,609 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c2bcbbf42dd249487e9caccf35ab2b80, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:56,609 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088056609"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088056609"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088056609"}]},"ts":"1676088056609"} 2023-02-11 04:00:56,615 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c2bcbbf42dd249487e9caccf35ab2b80, server=jenkins-hbase4.apache.org,33927,1676088054347}] 2023-02-11 04:00:56,769 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:56,769 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:00:56,775 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34166, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:00:56,781 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,781 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c2bcbbf42dd249487e9caccf35ab2b80, NAME => 'hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:56,782 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,782 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:56,782 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,782 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,784 INFO [StoreOpener-c2bcbbf42dd249487e9caccf35ab2b80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,786 DEBUG [StoreOpener-c2bcbbf42dd249487e9caccf35ab2b80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/info 2023-02-11 04:00:56,786 DEBUG [StoreOpener-c2bcbbf42dd249487e9caccf35ab2b80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/info 2023-02-11 04:00:56,787 INFO [StoreOpener-c2bcbbf42dd249487e9caccf35ab2b80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2bcbbf42dd249487e9caccf35ab2b80 columnFamilyName info 2023-02-11 04:00:56,787 INFO [StoreOpener-c2bcbbf42dd249487e9caccf35ab2b80-1] regionserver.HStore(310): Store=c2bcbbf42dd249487e9caccf35ab2b80/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:56,789 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,790 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,794 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:56,797 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:56,798 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c2bcbbf42dd249487e9caccf35ab2b80; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72206385, jitterRate=0.0759589821100235}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:56,798 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c2bcbbf42dd249487e9caccf35ab2b80: Running coprocessor pre-open hook at 1676088056782Writing region info on filesystem at 1676088056782Initializing all the Stores at 1676088056784 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088056784Cleaning up temporary data from old regions at 1676088056791 (+7 ms)Cleaning up detritus from prior splits at 1676088056791Running coprocessor post-open hooks at 1676088056798 (+7 ms)Region opened successfully at 1676088056798 2023-02-11 04:00:56,800 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80., pid=6, masterSystemTime=1676088056769 2023-02-11 04:00:56,804 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,805 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:56,807 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c2bcbbf42dd249487e9caccf35ab2b80, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:56,807 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088056806"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088056806"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088056806"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088056806"}]},"ts":"1676088056806"} 2023-02-11 04:00:56,814 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 04:00:56,814 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c2bcbbf42dd249487e9caccf35ab2b80, server=jenkins-hbase4.apache.org,33927,1676088054347 in 195 msec 2023-02-11 04:00:56,817 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 04:00:56,818 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c2bcbbf42dd249487e9caccf35ab2b80, ASSIGN in 363 msec 2023-02-11 04:00:56,819 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 04:00:56,819 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088056819"}]},"ts":"1676088056819"} 2023-02-11 04:00:56,821 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 04:00:56,825 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 04:00:56,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 539 msec 2023-02-11 04:00:56,897 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 04:00:56,899 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:00:56,899 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:56,919 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:56,922 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34170, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:56,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 04:00:56,952 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:00:56,958 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-02-11 04:00:56,968 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 04:00:56,979 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:00:56,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-02-11 04:00:56,993 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 04:00:56,997 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 04:00:56,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.568sec 2023-02-11 04:00:57,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 04:00:57,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 04:00:57,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 04:00:57,003 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36021,1676088053177-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 04:00:57,005 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36021,1676088053177-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 04:00:57,005 DEBUG [Listener at localhost/41065] zookeeper.ReadOnlyZKClient(139): Connect 0x24d4d701 to 127.0.0.1:53335 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:57,010 DEBUG [Listener at localhost/41065] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f65fe1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:57,016 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 04:00:57,025 DEBUG [hconnection-0x6274e043-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:57,036 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44806, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:57,046 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:57,046 DEBUG [Listener at localhost/41065] zookeeper.ReadOnlyZKClient(139): Connect 0x509b497f to 127.0.0.1:53335 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:57,056 DEBUG [ReadOnlyZKClient-127.0.0.1:53335@0x509b497f] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75cff840, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:57,127 DEBUG [Listener at localhost/41065] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:00:57,130 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37504, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:00:57,131 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36649] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36649,1676088054304' ***** 2023-02-11 04:00:57,131 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36649] regionserver.HRegionServer(2310): STOPPED: Called by admin client org.apache.hadoop.hbase.client.AsyncConnectionImpl@61bce98f 2023-02-11 04:00:57,132 INFO [RS:0;jenkins-hbase4:36649] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:00:57,133 INFO [RS:0;jenkins-hbase4:36649] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:00:57,133 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:00:57,133 INFO [RS:0;jenkins-hbase4:36649] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:00:57,134 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:57,134 DEBUG [RS:0;jenkins-hbase4:36649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x422bdf92 to 127.0.0.1:53335 2023-02-11 04:00:57,134 DEBUG [RS:0;jenkins-hbase4:36649] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,134 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36649,1676088054304; all regions closed. 2023-02-11 04:00:57,136 DEBUG [Listener at localhost/41065] client.ConnectionUtils(586): Start fetching master stub from registry 2023-02-11 04:00:57,139 DEBUG [ReadOnlyZKClient-127.0.0.1:53335@0x509b497f] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:57,144 DEBUG [ReadOnlyZKClient-127.0.0.1:53335@0x509b497f] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@57bb0d6e 2023-02-11 04:00:57,150 DEBUG [ReadOnlyZKClient-127.0.0.1:53335@0x509b497f] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 04:00:57,153 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49166, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 04:00:57,154 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36021] master.MasterRpcServices(1601): Client=jenkins//172.31.14.131 stop 2023-02-11 04:00:57,154 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36021] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36021,1676088053177' ***** 2023-02-11 04:00:57,154 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36021] regionserver.HRegionServer(2310): STOPPED: Stopped by RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36021 2023-02-11 04:00:57,156 DEBUG [M:0;jenkins-hbase4:36021] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18a5b18f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:57,156 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36021,1676088053177 2023-02-11 04:00:57,156 DEBUG [RS:0;jenkins-hbase4:36649] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs 2023-02-11 04:00:57,157 INFO [RS:0;jenkins-hbase4:36649] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36649%2C1676088054304:(num 1676088055760) 2023-02-11 04:00:57,157 DEBUG [RS:0;jenkins-hbase4:36649] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,158 DEBUG [M:0;jenkins-hbase4:36021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d72e187 to 127.0.0.1:53335 2023-02-11 04:00:57,158 DEBUG [M:0;jenkins-hbase4:36021] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,158 INFO [RS:0;jenkins-hbase4:36649] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:57,159 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36021,1676088053177; all regions closed. 2023-02-11 04:00:57,159 DEBUG [M:0;jenkins-hbase4:36021] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,159 DEBUG [M:0;jenkins-hbase4:36021] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 04:00:57,159 INFO [RS:0;jenkins-hbase4:36649] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 04:00:57,160 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 04:00:57,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088055436] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088055436,5,FailOnTimeoutGroup] 2023-02-11 04:00:57,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088055440] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088055440,5,FailOnTimeoutGroup] 2023-02-11 04:00:57,160 DEBUG [M:0;jenkins-hbase4:36021] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 04:00:57,160 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:57,160 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:57,160 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:57,160 INFO [M:0;jenkins-hbase4:36021] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 04:00:57,161 INFO [M:0;jenkins-hbase4:36021] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 04:00:57,160 INFO [RS:0;jenkins-hbase4:36649] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:00:57,161 INFO [M:0;jenkins-hbase4:36021] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 04:00:57,161 INFO [RS:0;jenkins-hbase4:36649] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:00:57,160 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:57,160 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:57,162 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:57,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:57,162 INFO [RS:0;jenkins-hbase4:36649] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:00:57,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:57,162 DEBUG [M:0;jenkins-hbase4:36021] master.HMaster(1502): Stopping service threads 2023-02-11 04:00:57,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:57,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:57,163 INFO [M:0;jenkins-hbase4:36021] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 04:00:57,163 INFO [Listener at localhost/41065] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/41065. 2023-02-11 04:00:57,164 INFO [M:0;jenkins-hbase4:36021] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 04:00:57,164 INFO [RS:0;jenkins-hbase4:36649] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36649 2023-02-11 04:00:57,164 DEBUG [Listener at localhost/41065] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2.tearDown(TestAsyncClusterAdminApi2.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:57,165 ERROR [M:0;jenkins-hbase4:36021] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] 2023-02-11 04:00:57,165 INFO [M:0;jenkins-hbase4:36021] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 04:00:57,166 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 04:00:57,166 DEBUG [M:0;jenkins-hbase4:36021] zookeeper.ZKUtil(398): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 04:00:57,166 WARN [M:0;jenkins-hbase4:36021] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 04:00:57,166 INFO [M:0;jenkins-hbase4:36021] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 04:00:57,167 INFO [M:0;jenkins-hbase4:36021] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 04:00:57,168 DEBUG [M:0;jenkins-hbase4:36021] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 04:00:57,168 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:57,168 DEBUG [M:0;jenkins-hbase4:36021] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:57,168 DEBUG [M:0;jenkins-hbase4:36021] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 04:00:57,168 DEBUG [M:0;jenkins-hbase4:36021] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:57,169 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.07 KB heapSize=29.55 KB 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:57,175 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:57,175 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@10a443bc rejected from java.util.concurrent.ThreadPoolExecutor@149bc570[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:57,176 DEBUG [Listener at localhost/41065] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,177 DEBUG [Listener at localhost/41065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x509b497f to 127.0.0.1:53335 2023-02-11 04:00:57,177 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 04:00:57,177 DEBUG [Listener at localhost/41065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24d4d701 to 127.0.0.1:53335 2023-02-11 04:00:57,177 DEBUG [Listener at localhost/41065] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,179 DEBUG [Listener at localhost/41065] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 04:00:57,179 INFO [Listener at localhost/41065] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33927,1676088054347' ***** 2023-02-11 04:00:57,179 INFO [Listener at localhost/41065] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 04:00:57,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:57,179 INFO [Listener at localhost/41065] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43901,1676088054377' ***** 2023-02-11 04:00:57,179 INFO [RS:1;jenkins-hbase4:33927] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:00:57,179 INFO [Listener at localhost/41065] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 04:00:57,179 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:00:57,179 INFO [RS:1;jenkins-hbase4:33927] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:00:57,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:57,180 INFO [RS:1;jenkins-hbase4:33927] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:00:57,180 INFO [RS:2;jenkins-hbase4:43901] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:00:57,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:57,181 INFO [RS:2;jenkins-hbase4:43901] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:00:57,181 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 znode expired, triggering replicatorRemoved event 2023-02-11 04:00:57,181 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,36649,1676088054304 znode expired, triggering replicatorRemoved event 2023-02-11 04:00:57,181 INFO [RS:2;jenkins-hbase4:43901] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:00:57,181 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(3304): Received CLOSE for c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:57,181 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:00:57,182 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:57,183 DEBUG [RS:2;jenkins-hbase4:43901] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f39620d to 127.0.0.1:53335 2023-02-11 04:00:57,183 DEBUG [RS:2;jenkins-hbase4:43901] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,183 INFO [RS:2;jenkins-hbase4:43901] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:00:57,183 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:57,183 INFO [RS:2;jenkins-hbase4:43901] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:00:57,184 DEBUG [RS:1;jenkins-hbase4:33927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49f65a87 to 127.0.0.1:53335 2023-02-11 04:00:57,184 INFO [RS:2;jenkins-hbase4:43901] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:00:57,184 DEBUG [RS:1;jenkins-hbase4:33927] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c2bcbbf42dd249487e9caccf35ab2b80, disabling compactions & flushes 2023-02-11 04:00:57,184 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-11 04:00:57,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:57,184 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 04:00:57,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:57,185 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1479): Online Regions={c2bcbbf42dd249487e9caccf35ab2b80=hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80.} 2023-02-11 04:00:57,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. after waiting 0 ms 2023-02-11 04:00:57,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:57,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c2bcbbf42dd249487e9caccf35ab2b80 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 04:00:57,186 DEBUG [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1505): Waiting on c2bcbbf42dd249487e9caccf35ab2b80 2023-02-11 04:00:57,186 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-11 04:00:57,187 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-11 04:00:57,187 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 04:00:57,187 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-11 04:00:57,187 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 04:00:57,188 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 04:00:57,188 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 04:00:57,188 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 04:00:57,188 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-02-11 04:00:57,207 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:57,209 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:57,212 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:57,266 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/.tmp/info/bba6045dfddb4a6ab5433f4af51fb829 2023-02-11 04:00:57,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/.tmp/info/2af10f949ca744ef9f6a2e3b792ce1c9 2023-02-11 04:00:57,268 INFO [M:0;jenkins-hbase4:36021] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2fbd5e43fab745c39af649f8f29e853a 2023-02-11 04:00:57,312 DEBUG [M:0;jenkins-hbase4:36021] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2fbd5e43fab745c39af649f8f29e853a as hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2fbd5e43fab745c39af649f8f29e853a 2023-02-11 04:00:57,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/.tmp/info/2af10f949ca744ef9f6a2e3b792ce1c9 as hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/info/2af10f949ca744ef9f6a2e3b792ce1c9 2023-02-11 04:00:57,324 INFO [M:0;jenkins-hbase4:36021] regionserver.HStore(1080): Added hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2fbd5e43fab745c39af649f8f29e853a, entries=8, sequenceid=66, filesize=6.3 K 2023-02-11 04:00:57,325 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/info/2af10f949ca744ef9f6a2e3b792ce1c9, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 04:00:57,326 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegion(2947): Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 157ms, sequenceid=66, compaction requested=false 2023-02-11 04:00:57,332 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:57,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c2bcbbf42dd249487e9caccf35ab2b80 in 147ms, sequenceid=6, compaction requested=false 2023-02-11 04:00:57,332 DEBUG [M:0;jenkins-hbase4:36021] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676088057168Disabling compacts and flushes for region at 1676088057168Disabling writes for close at 1676088057168Obtaining lock to block concurrent updates at 1676088057169 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676088057169Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=24646, getHeapSize=30248, getOffHeapSize=0, getCellsCount=71 at 1676088057185 (+16 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676088057188 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676088057190 (+2 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676088057221 (+31 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676088057224 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676088057314 (+90 ms)Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 157ms, sequenceid=66, compaction requested=false at 1676088057326 (+12 ms)Writing region close event to WAL at 1676088057332 (+6 ms)Closed at 1676088057332 2023-02-11 04:00:57,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-11 04:00:57,346 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:57,346 INFO [M:0;jenkins-hbase4:36021] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 04:00:57,347 INFO [M:0;jenkins-hbase4:36021] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36021 2023-02-11 04:00:57,350 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/.tmp/table/0f8e35ec4d124887ac5b5bda28ec914a 2023-02-11 04:00:57,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/namespace/c2bcbbf42dd249487e9caccf35ab2b80/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 04:00:57,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:57,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c2bcbbf42dd249487e9caccf35ab2b80: Waiting for close lock at 1676088057184Running coprocessor pre-close hooks at 1676088057184Disabling compacts and flushes for region at 1676088057184Disabling writes for close at 1676088057185 (+1 ms)Obtaining lock to block concurrent updates at 1676088057185Preparing flush snapshotting stores in c2bcbbf42dd249487e9caccf35ab2b80 at 1676088057185Finished memstore snapshotting hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676088057185Flushing stores of hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. at 1676088057187 (+2 ms)Flushing c2bcbbf42dd249487e9caccf35ab2b80/info: creating writer at 1676088057190 (+3 ms)Flushing c2bcbbf42dd249487e9caccf35ab2b80/info: appending metadata at 1676088057220 (+30 ms)Flushing c2bcbbf42dd249487e9caccf35ab2b80/info: closing flushed file at 1676088057224 (+4 ms)Flushing c2bcbbf42dd249487e9caccf35ab2b80/info: reopening flushed file at 1676088057315 (+91 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c2bcbbf42dd249487e9caccf35ab2b80 in 147ms, sequenceid=6, compaction requested=false at 1676088057332 (+17 ms)Writing region close event to WAL at 1676088057342 (+10 ms)Running coprocessor post-close hooks at 1676088057355 (+13 ms)Closed at 1676088057355 2023-02-11 04:00:57,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676088056284.c2bcbbf42dd249487e9caccf35ab2b80. 2023-02-11 04:00:57,357 DEBUG [M:0;jenkins-hbase4:36021] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,36021,1676088053177 already deleted, retry=false 2023-02-11 04:00:57,360 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/.tmp/info/bba6045dfddb4a6ab5433f4af51fb829 as hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/info/bba6045dfddb4a6ab5433f4af51fb829 2023-02-11 04:00:57,368 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/info/bba6045dfddb4a6ab5433f4af51fb829, entries=10, sequenceid=9, filesize=5.9 K 2023-02-11 04:00:57,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/.tmp/table/0f8e35ec4d124887ac5b5bda28ec914a as hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/table/0f8e35ec4d124887ac5b5bda28ec914a 2023-02-11 04:00:57,382 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/table/0f8e35ec4d124887ac5b5bda28ec914a, entries=2, sequenceid=9, filesize=4.7 K 2023-02-11 04:00:57,384 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 196ms, sequenceid=9, compaction requested=false 2023-02-11 04:00:57,384 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-11 04:00:57,386 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33927,1676088054347; all regions closed. 2023-02-11 04:00:57,388 DEBUG [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-11 04:00:57,397 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-02-11 04:00:57,397 DEBUG [RS:1;jenkins-hbase4:33927] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs 2023-02-11 04:00:57,397 INFO [RS:1;jenkins-hbase4:33927] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33927%2C1676088054347:(num 1676088055760) 2023-02-11 04:00:57,397 DEBUG [RS:1;jenkins-hbase4:33927] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,397 INFO [RS:1;jenkins-hbase4:33927] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:57,397 INFO [RS:1;jenkins-hbase4:33927] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 04:00:57,398 INFO [RS:1;jenkins-hbase4:33927] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:00:57,398 INFO [RS:1;jenkins-hbase4:33927] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:00:57,398 INFO [RS:1;jenkins-hbase4:33927] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:00:57,398 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 04:00:57,398 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:57,399 INFO [RS:1;jenkins-hbase4:33927] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33927 2023-02-11 04:00:57,399 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:57,402 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676088057187Running coprocessor pre-close hooks at 1676088057187Disabling compacts and flushes for region at 1676088057187Disabling writes for close at 1676088057188 (+1 ms)Obtaining lock to block concurrent updates at 1676088057188Preparing flush snapshotting stores in 1588230740 at 1676088057188Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1290, getHeapSize=2912, getOffHeapSize=0, getCellsCount=12 at 1676088057189 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676088057189Flushing 1588230740/info: creating writer at 1676088057190 (+1 ms)Flushing 1588230740/info: appending metadata at 1676088057220 (+30 ms)Flushing 1588230740/info: closing flushed file at 1676088057224 (+4 ms)Flushing 1588230740/table: creating writer at 1676088057311 (+87 ms)Flushing 1588230740/table: appending metadata at 1676088057316 (+5 ms)Flushing 1588230740/table: closing flushed file at 1676088057316Flushing 1588230740/info: reopening flushed file at 1676088057361 (+45 ms)Flushing 1588230740/table: reopening flushed file at 1676088057370 (+9 ms)Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 196ms, sequenceid=9, compaction requested=false at 1676088057384 (+14 ms)Writing region close event to WAL at 1676088057389 (+5 ms)Running coprocessor post-close hooks at 1676088057398 (+9 ms)Closed at 1676088057399 (+1 ms) 2023-02-11 04:00:57,403 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:57,404 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:57,404 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33927,1676088054347 2023-02-11 04:00:57,405 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c4939ea rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,405 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:57,405 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:57,405 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@39b04523 rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,589 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43901,1676088054377; all regions closed. 2023-02-11 04:00:57,596 DEBUG [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs 2023-02-11 04:00:57,596 INFO [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43901%2C1676088054377.meta:.meta(num 1676088056018) 2023-02-11 04:00:57,603 DEBUG [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/oldWALs 2023-02-11 04:00:57,603 INFO [RS:2;jenkins-hbase4:43901] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43901%2C1676088054377:(num 1676088055760) 2023-02-11 04:00:57,603 DEBUG [RS:2;jenkins-hbase4:43901] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:57,603 INFO [RS:2;jenkins-hbase4:43901] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:57,603 INFO [RS:2;jenkins-hbase4:43901] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 04:00:57,603 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:57,604 INFO [RS:2;jenkins-hbase4:43901] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43901 2023-02-11 04:00:57,607 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43901,1676088054377 2023-02-11 04:00:57,607 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@15477b1d rejected from java.util.concurrent.ThreadPoolExecutor@1178ed6d[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,739 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:57,739 INFO [RS:0;jenkins-hbase4:36649] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36649,1676088054304; zookeeper connection closed. 2023-02-11 04:00:57,739 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5209fe1a rejected from java.util.concurrent.ThreadPoolExecutor@149bc570[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,740 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x1004189f4230001, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:57,740 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@46241cc7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@46241cc7 2023-02-11 04:00:57,740 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2af81761 rejected from java.util.concurrent.ThreadPoolExecutor@149bc570[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,940 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:57,940 INFO [RS:2;jenkins-hbase4:43901] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43901,1676088054377; zookeeper connection closed. 2023-02-11 04:00:57,940 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4bd5b5da rejected from java.util.concurrent.ThreadPoolExecutor@1178ed6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:57,940 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@db47c5b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@db47c5b 2023-02-11 04:00:57,940 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:43901-0x1004189f4230003, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:57,940 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@33a13a7a rejected from java.util.concurrent.ThreadPoolExecutor@1178ed6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:58,040 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:58,040 INFO [RS:1;jenkins-hbase4:33927] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33927,1676088054347; zookeeper connection closed. 2023-02-11 04:00:58,040 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@419d95c7 rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:58,040 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): regionserver:33927-0x1004189f4230002, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:58,040 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@10d4ce2a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@10d4ce2a 2023-02-11 04:00:58,040 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@56f9054b rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:58,041 INFO [Listener at localhost/41065] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 04:00:58,140 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:58,140 INFO [M:0;jenkins-hbase4:36021] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36021,1676088053177; zookeeper connection closed. 2023-02-11 04:00:58,140 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63a79bfa rejected from java.util.concurrent.ThreadPoolExecutor@18cff1a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:58,140 DEBUG [Listener at localhost/41065-EventThread] zookeeper.ZKWatcher(600): master:36021-0x1004189f4230000, quorum=127.0.0.1:53335, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:58,140 ERROR [Listener at localhost/41065-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b49dd15 rejected from java.util.concurrent.ThreadPoolExecutor@18cff1a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:58,142 WARN [Listener at localhost/41065] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:00:58,145 INFO [Listener at localhost/41065] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:58,247 WARN [BP-392444252-172.31.14.131-1676088050078 heartbeating to localhost/127.0.0.1:42141] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:00:58,247 WARN [BP-392444252-172.31.14.131-1676088050078 heartbeating to localhost/127.0.0.1:42141] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-392444252-172.31.14.131-1676088050078 (Datanode Uuid 84c323d8-9611-48a9-996c-60ff152a6d06) service to localhost/127.0.0.1:42141 2023-02-11 04:00:58,249 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/dfs/data/data5/current/BP-392444252-172.31.14.131-1676088050078] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:58,249 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/dfs/data/data6/current/BP-392444252-172.31.14.131-1676088050078] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:58,249 WARN [Listener at localhost/41065] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:00:58,251 INFO [Listener at localhost/41065] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:58,354 WARN [BP-392444252-172.31.14.131-1676088050078 heartbeating to localhost/127.0.0.1:42141] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:00:58,354 WARN [BP-392444252-172.31.14.131-1676088050078 heartbeating to localhost/127.0.0.1:42141] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-392444252-172.31.14.131-1676088050078 (Datanode Uuid 5dcf6bcd-f24d-4994-8ecb-5ee6400ae0c8) service to localhost/127.0.0.1:42141 2023-02-11 04:00:58,355 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/dfs/data/data3/current/BP-392444252-172.31.14.131-1676088050078] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:58,355 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/dfs/data/data4/current/BP-392444252-172.31.14.131-1676088050078] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:58,355 WARN [Listener at localhost/41065] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:00:58,357 INFO [Listener at localhost/41065] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:58,459 WARN [BP-392444252-172.31.14.131-1676088050078 heartbeating to localhost/127.0.0.1:42141] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:00:58,459 WARN [BP-392444252-172.31.14.131-1676088050078 heartbeating to localhost/127.0.0.1:42141] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-392444252-172.31.14.131-1676088050078 (Datanode Uuid 93c9d8c5-f8b2-4201-b019-dba9d527bdb7) service to localhost/127.0.0.1:42141 2023-02-11 04:00:58,460 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/dfs/data/data1/current/BP-392444252-172.31.14.131-1676088050078] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:58,460 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/cluster_4c6b08fa-e8ab-3b4d-2b39-c2a509d8cc01/dfs/data/data2/current/BP-392444252-172.31.14.131-1676088050078] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:58,488 INFO [Listener at localhost/41065] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:58,601 INFO [Listener at localhost/41065] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 04:00:58,639 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-11 04:00:58,653 INFO [Listener at localhost/41065] hbase.ResourceChecker(175): after: client.TestAsyncClusterAdminApi2#testStop Thread=79 (was 8) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:42141 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:42141 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:42141 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:42141 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: Listener at localhost/41065 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:42141 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:42141 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@530ebddf java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'NameNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:42141 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:42141 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:42141 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=504 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=261 (was 229) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3147 (was 3723) 2023-02-11 04:00:58,663 INFO [Listener at localhost/41065] hbase.ResourceChecker(147): before: client.TestAsyncClusterAdminApi2#testShutdown Thread=79, OpenFileDescriptor=504, MaxFileDescriptor=60000, SystemLoadAverage=261, ProcessCount=169, AvailableMemoryMB=3146 2023-02-11 04:00:58,663 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 04:00:58,664 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/hadoop.log.dir so I do NOT create it in target/test-data/0566c872-dd62-4bae-a948-556e214a765d 2023-02-11 04:00:58,664 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7b21381d-3e3e-84de-09a4-5a6cf8f73929/hadoop.tmp.dir so I do NOT create it in target/test-data/0566c872-dd62-4bae-a948-556e214a765d 2023-02-11 04:00:58,664 INFO [Listener at localhost/41065] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6, deleteOnExit=true 2023-02-11 04:00:58,664 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 04:00:58,665 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/test.cache.data in system properties and HBase conf 2023-02-11 04:00:58,665 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 04:00:58,665 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/hadoop.log.dir in system properties and HBase conf 2023-02-11 04:00:58,665 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 04:00:58,665 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 04:00:58,666 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 04:00:58,666 DEBUG [Listener at localhost/41065] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 04:00:58,666 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 04:00:58,666 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 04:00:58,667 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 04:00:58,667 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 04:00:58,667 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 04:00:58,667 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 04:00:58,668 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 04:00:58,668 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 04:00:58,668 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 04:00:58,668 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/nfs.dump.dir in system properties and HBase conf 2023-02-11 04:00:58,668 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/java.io.tmpdir in system properties and HBase conf 2023-02-11 04:00:58,669 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 04:00:58,669 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 04:00:58,669 INFO [Listener at localhost/41065] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 04:00:58,674 WARN [Listener at localhost/41065] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 04:00:58,674 WARN [Listener at localhost/41065] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 04:00:58,720 WARN [Listener at localhost/41065] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:58,723 INFO [Listener at localhost/41065] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:58,729 INFO [Listener at localhost/41065] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/java.io.tmpdir/Jetty_localhost_39207_hdfs____d15i65/webapp 2023-02-11 04:00:58,826 INFO [Listener at localhost/41065] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39207 2023-02-11 04:00:58,830 WARN [Listener at localhost/41065] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 04:00:58,831 WARN [Listener at localhost/41065] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 04:00:58,874 WARN [Listener at localhost/41045] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:58,885 WARN [Listener at localhost/41045] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:58,887 WARN [Listener at localhost/41045] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:58,889 INFO [Listener at localhost/41045] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:58,893 INFO [Listener at localhost/41045] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/java.io.tmpdir/Jetty_localhost_39691_datanode____.fqmyr5/webapp 2023-02-11 04:00:58,986 INFO [Listener at localhost/41045] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39691 2023-02-11 04:00:58,995 WARN [Listener at localhost/33337] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:59,013 WARN [Listener at localhost/33337] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:59,015 WARN [Listener at localhost/33337] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:59,016 INFO [Listener at localhost/33337] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:59,022 INFO [Listener at localhost/33337] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/java.io.tmpdir/Jetty_localhost_37733_datanode____.4hny5i/webapp 2023-02-11 04:00:59,099 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2b9aadd23584e36b: Processing first storage report for DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b from datanode ba3e46fc-0706-4b4a-ad86-8f7bbbeb9807 2023-02-11 04:00:59,100 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2b9aadd23584e36b: from storage DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b node DatanodeRegistration(127.0.0.1:38521, datanodeUuid=ba3e46fc-0706-4b4a-ad86-8f7bbbeb9807, infoPort=34181, infoSecurePort=0, ipcPort=33337, storageInfo=lv=-57;cid=testClusterID;nsid=1596860647;c=1676088058677), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 04:00:59,100 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2b9aadd23584e36b: Processing first storage report for DS-e620981f-7482-43ca-a04a-22b5fa38870b from datanode ba3e46fc-0706-4b4a-ad86-8f7bbbeb9807 2023-02-11 04:00:59,100 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2b9aadd23584e36b: from storage DS-e620981f-7482-43ca-a04a-22b5fa38870b node DatanodeRegistration(127.0.0.1:38521, datanodeUuid=ba3e46fc-0706-4b4a-ad86-8f7bbbeb9807, infoPort=34181, infoSecurePort=0, ipcPort=33337, storageInfo=lv=-57;cid=testClusterID;nsid=1596860647;c=1676088058677), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:59,136 INFO [Listener at localhost/33337] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37733 2023-02-11 04:00:59,144 WARN [Listener at localhost/36555] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:59,162 WARN [Listener at localhost/36555] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:59,164 WARN [Listener at localhost/36555] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:59,166 INFO [Listener at localhost/36555] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:59,170 INFO [Listener at localhost/36555] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/java.io.tmpdir/Jetty_localhost_37703_datanode____klbofx/webapp 2023-02-11 04:00:59,242 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9af40f9e247c51bf: Processing first storage report for DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e from datanode 042d5c25-1621-492b-8e44-10f3ca3f3485 2023-02-11 04:00:59,242 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9af40f9e247c51bf: from storage DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e node DatanodeRegistration(127.0.0.1:36549, datanodeUuid=042d5c25-1621-492b-8e44-10f3ca3f3485, infoPort=39541, infoSecurePort=0, ipcPort=36555, storageInfo=lv=-57;cid=testClusterID;nsid=1596860647;c=1676088058677), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:59,242 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9af40f9e247c51bf: Processing first storage report for DS-adf6ff75-2c28-44d6-9322-b544475eb34a from datanode 042d5c25-1621-492b-8e44-10f3ca3f3485 2023-02-11 04:00:59,242 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9af40f9e247c51bf: from storage DS-adf6ff75-2c28-44d6-9322-b544475eb34a node DatanodeRegistration(127.0.0.1:36549, datanodeUuid=042d5c25-1621-492b-8e44-10f3ca3f3485, infoPort=39541, infoSecurePort=0, ipcPort=36555, storageInfo=lv=-57;cid=testClusterID;nsid=1596860647;c=1676088058677), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:59,277 INFO [Listener at localhost/36555] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37703 2023-02-11 04:00:59,285 WARN [Listener at localhost/33533] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:59,394 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xda574f2c938753ca: Processing first storage report for DS-43a84765-a839-43d9-8bef-247d1fdacafc from datanode 2124e074-f7f0-440b-b0fb-3bf35e8e3e5e 2023-02-11 04:00:59,394 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xda574f2c938753ca: from storage DS-43a84765-a839-43d9-8bef-247d1fdacafc node DatanodeRegistration(127.0.0.1:41997, datanodeUuid=2124e074-f7f0-440b-b0fb-3bf35e8e3e5e, infoPort=42307, infoSecurePort=0, ipcPort=33533, storageInfo=lv=-57;cid=testClusterID;nsid=1596860647;c=1676088058677), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 04:00:59,394 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xda574f2c938753ca: Processing first storage report for DS-004cf0f4-50eb-44a5-bd18-ff983199ac1e from datanode 2124e074-f7f0-440b-b0fb-3bf35e8e3e5e 2023-02-11 04:00:59,394 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xda574f2c938753ca: from storage DS-004cf0f4-50eb-44a5-bd18-ff983199ac1e node DatanodeRegistration(127.0.0.1:41997, datanodeUuid=2124e074-f7f0-440b-b0fb-3bf35e8e3e5e, infoPort=42307, infoSecurePort=0, ipcPort=33533, storageInfo=lv=-57;cid=testClusterID;nsid=1596860647;c=1676088058677), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:59,496 DEBUG [Listener at localhost/33533] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d 2023-02-11 04:00:59,502 INFO [Listener at localhost/33533] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/zookeeper_0, clientPort=51435, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 04:00:59,503 INFO [Listener at localhost/33533] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51435 2023-02-11 04:00:59,504 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,505 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,526 INFO [Listener at localhost/33533] util.FSUtils(479): Created version file at hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1 with version=8 2023-02-11 04:00:59,527 INFO [Listener at localhost/33533] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:42141/user/jenkins/test-data/374f6a33-4e5c-d1b4-4839-446f259cc4f8/hbase-staging 2023-02-11 04:00:59,528 INFO [Listener at localhost/33533] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:59,529 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,529 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,529 INFO [Listener at localhost/33533] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:59,529 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,529 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:59,529 INFO [Listener at localhost/33533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:59,531 INFO [Listener at localhost/33533] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45001 2023-02-11 04:00:59,531 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,533 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,534 INFO [Listener at localhost/33533] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45001 connecting to ZooKeeper ensemble=127.0.0.1:51435 2023-02-11 04:00:59,545 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:450010x0, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:59,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:45001-0x100418a102f0000 connected 2023-02-11 04:00:59,567 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:59,567 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:59,568 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:59,568 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45001 2023-02-11 04:00:59,572 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45001 2023-02-11 04:00:59,572 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45001 2023-02-11 04:00:59,572 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45001 2023-02-11 04:00:59,573 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45001 2023-02-11 04:00:59,573 INFO [Listener at localhost/33533] master.HMaster(439): hbase.rootdir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1, hbase.cluster.distributed=false 2023-02-11 04:00:59,590 INFO [Listener at localhost/33533] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:59,590 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,590 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,590 INFO [Listener at localhost/33533] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:59,591 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,591 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:59,591 INFO [Listener at localhost/33533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:59,592 INFO [Listener at localhost/33533] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43137 2023-02-11 04:00:59,593 INFO [Listener at localhost/33533] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:59,594 DEBUG [Listener at localhost/33533] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 04:00:59,595 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,596 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,597 INFO [Listener at localhost/33533] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43137 connecting to ZooKeeper ensemble=127.0.0.1:51435 2023-02-11 04:00:59,600 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:431370x0, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:59,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43137-0x100418a102f0001 connected 2023-02-11 04:00:59,601 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:59,602 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:59,603 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:59,607 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43137 2023-02-11 04:00:59,607 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43137 2023-02-11 04:00:59,607 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43137 2023-02-11 04:00:59,607 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43137 2023-02-11 04:00:59,608 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43137 2023-02-11 04:00:59,619 INFO [Listener at localhost/33533] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:59,619 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,619 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,619 INFO [Listener at localhost/33533] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:59,619 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,619 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:59,620 INFO [Listener at localhost/33533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:59,621 INFO [Listener at localhost/33533] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36163 2023-02-11 04:00:59,622 INFO [Listener at localhost/33533] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:59,623 DEBUG [Listener at localhost/33533] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 04:00:59,623 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,625 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,626 INFO [Listener at localhost/33533] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36163 connecting to ZooKeeper ensemble=127.0.0.1:51435 2023-02-11 04:00:59,630 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:361630x0, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:59,630 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:361630x0, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:59,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36163-0x100418a102f0002 connected 2023-02-11 04:00:59,631 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:59,632 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:59,632 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36163 2023-02-11 04:00:59,632 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36163 2023-02-11 04:00:59,633 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36163 2023-02-11 04:00:59,636 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36163 2023-02-11 04:00:59,636 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36163 2023-02-11 04:00:59,647 INFO [Listener at localhost/33533] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 04:00:59,647 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,647 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,647 INFO [Listener at localhost/33533] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:59,647 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:59,648 INFO [Listener at localhost/33533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:59,648 INFO [Listener at localhost/33533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:59,649 INFO [Listener at localhost/33533] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36477 2023-02-11 04:00:59,649 INFO [Listener at localhost/33533] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:59,650 DEBUG [Listener at localhost/33533] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 04:00:59,651 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,652 INFO [Listener at localhost/33533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,653 INFO [Listener at localhost/33533] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36477 connecting to ZooKeeper ensemble=127.0.0.1:51435 2023-02-11 04:00:59,656 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:364770x0, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:59,656 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:364770x0, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:59,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36477-0x100418a102f0003 connected 2023-02-11 04:00:59,657 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:59,658 DEBUG [Listener at localhost/33533] zookeeper.ZKUtil(164): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:59,660 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36477 2023-02-11 04:00:59,661 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36477 2023-02-11 04:00:59,661 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36477 2023-02-11 04:00:59,662 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36477 2023-02-11 04:00:59,662 DEBUG [Listener at localhost/33533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36477 2023-02-11 04:00:59,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,666 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 04:00:59,666 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,667 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:59,667 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:59,667 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:59,667 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:59,668 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:59,670 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 04:00:59,671 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45001,1676088059528 from backup master directory 2023-02-11 04:00:59,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 04:00:59,672 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,672 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 04:00:59,672 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:59,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/hbase.id with ID: 4f694e13-1609-454f-850a-4104a80d43e6 2023-02-11 04:00:59,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:59,712 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:59,727 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa7d30b to 127.0.0.1:51435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:59,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24260d5f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:59,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:59,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 04:00:59,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:59,736 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store-tmp 2023-02-11 04:00:59,751 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:59,751 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 04:00:59,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:59,751 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:59,752 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 04:00:59,752 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:59,752 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:59,752 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676088059751Disabling compacts and flushes for region at 1676088059751Disabling writes for close at 1676088059752 (+1 ms)Writing region close event to WAL at 1676088059752Closed at 1676088059752 2023-02-11 04:00:59,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/WALs/jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45001%2C1676088059528, suffix=, logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/WALs/jenkins-hbase4.apache.org,45001,1676088059528, archiveDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/oldWALs, maxLogs=10 2023-02-11 04:00:59,772 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK] 2023-02-11 04:00:59,773 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK] 2023-02-11 04:00:59,774 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK] 2023-02-11 04:00:59,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/WALs/jenkins-hbase4.apache.org,45001,1676088059528/jenkins-hbase4.apache.org%2C45001%2C1676088059528.1676088059757 2023-02-11 04:00:59,778 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK], DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK], DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK]] 2023-02-11 04:00:59,778 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:59,778 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:59,779 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:59,779 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:59,781 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:59,783 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 04:00:59,783 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 04:00:59,784 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:59,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:59,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:59,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:59,792 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:59,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66396978, jitterRate=-0.010607928037643433}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:59,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676088059779Initializing all the Stores at 1676088059780 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088059780Cleaning up temporary data from old regions at 1676088059786 (+6 ms)Cleaning up detritus from prior splits at 1676088059786Region opened successfully at 1676088059793 (+7 ms) 2023-02-11 04:00:59,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 04:00:59,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 04:00:59,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 04:00:59,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 04:00:59,795 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-11 04:00:59,795 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-11 04:00:59,795 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 04:00:59,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 04:00:59,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 04:00:59,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 04:00:59,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 04:00:59,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 04:00:59,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 04:00:59,808 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 04:00:59,810 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:59,810 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 04:00:59,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 04:00:59,812 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 04:00:59,813 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:59,813 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:59,813 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:59,813 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:59,813 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:59,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,45001,1676088059528, sessionid=0x100418a102f0000, setting cluster-up flag (Was=false) 2023-02-11 04:00:59,819 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:59,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 04:00:59,824 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,833 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:59,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 04:00:59,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:00:59,840 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/.hbase-snapshot/.tmp 2023-02-11 04:00:59,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:59,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:59,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676088089845 2023-02-11 04:00:59,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 04:00:59,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 04:00:59,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 04:00:59,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 04:00:59,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 04:00:59,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 04:00:59,847 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:59,847 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 04:00:59,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 04:00:59,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 04:00:59,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 04:00:59,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 04:00:59,849 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:59,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 04:00:59,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088059849,5,FailOnTimeoutGroup] 2023-02-11 04:00:59,854 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088059853,5,FailOnTimeoutGroup] 2023-02-11 04:00:59,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 04:00:59,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,865 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(952): ClusterId : 4f694e13-1609-454f-850a-4104a80d43e6 2023-02-11 04:00:59,865 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(952): ClusterId : 4f694e13-1609-454f-850a-4104a80d43e6 2023-02-11 04:00:59,867 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:59,867 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:59,867 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(952): ClusterId : 4f694e13-1609-454f-850a-4104a80d43e6 2023-02-11 04:00:59,867 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:59,870 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:59,870 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:59,871 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:59,871 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:59,871 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:59,871 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:59,872 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1 2023-02-11 04:00:59,872 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:59,872 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:59,874 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:59,876 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:59,877 DEBUG [RS:1;jenkins-hbase4:36163] zookeeper.ReadOnlyZKClient(139): Connect 0x59a12e0a to 127.0.0.1:51435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:59,877 DEBUG [RS:2;jenkins-hbase4:36477] zookeeper.ReadOnlyZKClient(139): Connect 0x015560e1 to 127.0.0.1:51435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:59,877 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:59,878 DEBUG [RS:0;jenkins-hbase4:43137] zookeeper.ReadOnlyZKClient(139): Connect 0x56861c35 to 127.0.0.1:51435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:59,884 DEBUG [RS:1;jenkins-hbase4:36163] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d9ea1e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:59,884 DEBUG [RS:2;jenkins-hbase4:36477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f929ebb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:59,884 DEBUG [RS:1;jenkins-hbase4:36163] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@189c2431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:59,884 DEBUG [RS:2;jenkins-hbase4:36477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bad0b91, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:59,890 DEBUG [RS:0;jenkins-hbase4:43137] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c7e7745, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:59,890 DEBUG [RS:0;jenkins-hbase4:43137] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@197ce2ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:59,902 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:36477 2023-02-11 04:00:59,902 INFO [RS:2;jenkins-hbase4:36477] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:59,902 INFO [RS:2;jenkins-hbase4:36477] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:59,902 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:59,903 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:36163 2023-02-11 04:00:59,903 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43137 2023-02-11 04:00:59,903 INFO [RS:1;jenkins-hbase4:36163] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:59,903 INFO [RS:1;jenkins-hbase4:36163] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:59,903 INFO [RS:0;jenkins-hbase4:43137] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:59,903 INFO [RS:0;jenkins-hbase4:43137] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:59,903 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:59,903 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:59,903 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45001,1676088059528 with isa=jenkins-hbase4.apache.org/172.31.14.131:36477, startcode=1676088059647 2023-02-11 04:00:59,904 DEBUG [RS:2;jenkins-hbase4:36477] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:59,904 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45001,1676088059528 with isa=jenkins-hbase4.apache.org/172.31.14.131:36163, startcode=1676088059618 2023-02-11 04:00:59,904 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45001,1676088059528 with isa=jenkins-hbase4.apache.org/172.31.14.131:43137, startcode=1676088059590 2023-02-11 04:00:59,904 DEBUG [RS:0;jenkins-hbase4:43137] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:59,905 DEBUG [RS:1;jenkins-hbase4:36163] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:59,905 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:59,911 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 04:00:59,911 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56803, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:59,913 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/info 2023-02-11 04:00:59,913 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 04:00:59,914 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:59,914 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 04:00:59,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45001] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:00:59,917 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1 2023-02-11 04:00:59,918 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:41045 2023-02-11 04:00:59,918 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:59,919 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46015, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:59,919 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36441, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:59,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45001] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:00:59,920 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45001] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:00:59,920 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:59,921 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1 2023-02-11 04:00:59,921 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:41045 2023-02-11 04:00:59,921 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1 2023-02-11 04:00:59,921 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:59,921 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 04:00:59,921 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:41045 2023-02-11 04:00:59,921 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:59,921 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:59,922 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:59,923 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 04:00:59,925 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/table 2023-02-11 04:00:59,925 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 04:00:59,926 DEBUG [RS:1;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:00:59,926 DEBUG [RS:0;jenkins-hbase4:43137] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:00:59,926 WARN [RS:1;jenkins-hbase4:36163] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:59,926 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43137,1676088059590] 2023-02-11 04:00:59,928 INFO [RS:1;jenkins-hbase4:36163] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:59,928 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36477,1676088059647] 2023-02-11 04:00:59,928 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36163,1676088059618] 2023-02-11 04:00:59,928 WARN [RS:0;jenkins-hbase4:43137] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:59,928 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1947): logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:00:59,929 INFO [RS:0;jenkins-hbase4:43137] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:59,929 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1947): logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:00:59,931 DEBUG [RS:2;jenkins-hbase4:36477] zookeeper.ZKUtil(162): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:00:59,931 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:59,931 WARN [RS:2;jenkins-hbase4:36477] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:59,931 INFO [RS:2;jenkins-hbase4:36477] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:59,931 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1947): logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:00:59,933 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740 2023-02-11 04:00:59,935 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740 2023-02-11 04:00:59,938 DEBUG [RS:1;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:00:59,938 DEBUG [RS:0;jenkins-hbase4:43137] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:00:59,938 DEBUG [RS:1;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:00:59,939 DEBUG [RS:2;jenkins-hbase4:36477] zookeeper.ZKUtil(162): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:00:59,939 DEBUG [RS:1;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:00:59,939 DEBUG [RS:0;jenkins-hbase4:43137] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:00:59,939 DEBUG [RS:2;jenkins-hbase4:36477] zookeeper.ZKUtil(162): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:00:59,940 DEBUG [RS:2;jenkins-hbase4:36477] zookeeper.ZKUtil(162): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:00:59,940 DEBUG [RS:0;jenkins-hbase4:43137] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:00:59,940 DEBUG [RS:1;jenkins-hbase4:36163] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:59,940 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 04:00:59,940 INFO [RS:1;jenkins-hbase4:36163] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:59,941 DEBUG [RS:2;jenkins-hbase4:36477] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:59,941 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:59,942 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 04:00:59,943 INFO [RS:0;jenkins-hbase4:43137] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:59,943 INFO [RS:2;jenkins-hbase4:36477] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:59,945 INFO [RS:1;jenkins-hbase4:36163] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:59,947 INFO [RS:0;jenkins-hbase4:43137] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:59,947 INFO [RS:2;jenkins-hbase4:36477] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:59,948 INFO [RS:1;jenkins-hbase4:36163] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:59,948 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,952 INFO [RS:2;jenkins-hbase4:36477] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:59,953 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:59,953 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:59,953 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,953 INFO [RS:0;jenkins-hbase4:43137] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:59,953 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,953 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:59,954 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72957998, jitterRate=0.08715888857841492}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 04:00:59,954 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676088059905Initializing all the Stores at 1676088059909 (+4 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088059909Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088059910 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088059911 (+1 ms)Cleaning up temporary data from old regions at 1676088059938 (+27 ms)Cleaning up detritus from prior splits at 1676088059939 (+1 ms)Region opened successfully at 1676088059954 (+15 ms) 2023-02-11 04:00:59,954 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 04:00:59,954 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 04:00:59,954 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 04:00:59,954 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 04:00:59,954 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 04:00:59,956 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:59,957 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,957 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:59,957 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,957 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676088059954Disabling compacts and flushes for region at 1676088059954Disabling writes for close at 1676088059954Writing region close event to WAL at 1676088059957 (+3 ms)Closed at 1676088059957 2023-02-11 04:00:59,957 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,957 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,957 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,957 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,957 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,957 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:59,957 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,958 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,958 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:1;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:59,959 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 04:00:59,959 DEBUG [RS:0;jenkins-hbase4:43137] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 DEBUG [RS:2;jenkins-hbase4:36477] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:59,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 04:00:59,968 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 04:00:59,968 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,969 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,972 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,973 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,973 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,973 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,973 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,973 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,973 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,974 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:59,985 INFO [RS:0;jenkins-hbase4:43137] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:59,985 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43137,1676088059590-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,989 INFO [RS:1;jenkins-hbase4:36163] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:59,989 INFO [RS:2;jenkins-hbase4:36477] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:59,990 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36163,1676088059618-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:59,990 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36477,1676088059647-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:01:00,000 INFO [RS:0;jenkins-hbase4:43137] regionserver.Replication(203): jenkins-hbase4.apache.org,43137,1676088059590 started 2023-02-11 04:01:00,001 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43137,1676088059590, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43137, sessionid=0x100418a102f0001 2023-02-11 04:01:00,001 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:01:00,001 DEBUG [RS:0;jenkins-hbase4:43137] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:00,001 DEBUG [RS:0;jenkins-hbase4:43137] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43137,1676088059590' 2023-02-11 04:01:00,001 DEBUG [RS:0;jenkins-hbase4:43137] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:01:00,001 DEBUG [RS:0;jenkins-hbase4:43137] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:01:00,002 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:01:00,002 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:01:00,002 DEBUG [RS:0;jenkins-hbase4:43137] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:00,002 DEBUG [RS:0;jenkins-hbase4:43137] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43137,1676088059590' 2023-02-11 04:01:00,002 DEBUG [RS:0;jenkins-hbase4:43137] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:01:00,003 DEBUG [RS:0;jenkins-hbase4:43137] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:01:00,003 DEBUG [RS:0;jenkins-hbase4:43137] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:01:00,003 INFO [RS:0;jenkins-hbase4:43137] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:01:00,003 INFO [RS:0;jenkins-hbase4:43137] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:01:00,005 INFO [RS:2;jenkins-hbase4:36477] regionserver.Replication(203): jenkins-hbase4.apache.org,36477,1676088059647 started 2023-02-11 04:01:00,005 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36477,1676088059647, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36477, sessionid=0x100418a102f0003 2023-02-11 04:01:00,005 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:01:00,005 DEBUG [RS:2;jenkins-hbase4:36477] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:00,005 DEBUG [RS:2;jenkins-hbase4:36477] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36477,1676088059647' 2023-02-11 04:01:00,005 DEBUG [RS:2;jenkins-hbase4:36477] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:01:00,006 DEBUG [RS:2;jenkins-hbase4:36477] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:01:00,006 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:01:00,006 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:01:00,006 DEBUG [RS:2;jenkins-hbase4:36477] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:00,006 DEBUG [RS:2;jenkins-hbase4:36477] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36477,1676088059647' 2023-02-11 04:01:00,006 DEBUG [RS:2;jenkins-hbase4:36477] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:01:00,007 DEBUG [RS:2;jenkins-hbase4:36477] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:01:00,007 DEBUG [RS:2;jenkins-hbase4:36477] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:01:00,007 INFO [RS:2;jenkins-hbase4:36477] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:01:00,007 INFO [RS:2;jenkins-hbase4:36477] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:01:00,007 INFO [RS:1;jenkins-hbase4:36163] regionserver.Replication(203): jenkins-hbase4.apache.org,36163,1676088059618 started 2023-02-11 04:01:00,007 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36163,1676088059618, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36163, sessionid=0x100418a102f0002 2023-02-11 04:01:00,007 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:01:00,007 DEBUG [RS:1;jenkins-hbase4:36163] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:01:00,007 DEBUG [RS:1;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36163,1676088059618' 2023-02-11 04:01:00,007 DEBUG [RS:1;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36163,1676088059618' 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:01:00,008 DEBUG [RS:1;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:01:00,009 DEBUG [RS:1;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:01:00,009 INFO [RS:1;jenkins-hbase4:36163] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:01:00,009 INFO [RS:1;jenkins-hbase4:36163] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:01:00,106 INFO [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43137%2C1676088059590, suffix=, logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,43137,1676088059590, archiveDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs, maxLogs=32 2023-02-11 04:01:00,109 INFO [RS:2;jenkins-hbase4:36477] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36477%2C1676088059647, suffix=, logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36477,1676088059647, archiveDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs, maxLogs=32 2023-02-11 04:01:00,111 INFO [RS:1;jenkins-hbase4:36163] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36163%2C1676088059618, suffix=, logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36163,1676088059618, archiveDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs, maxLogs=32 2023-02-11 04:01:00,124 DEBUG [jenkins-hbase4:45001] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 04:01:00,127 DEBUG [jenkins-hbase4:45001] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:01:00,127 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK] 2023-02-11 04:01:00,128 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK] 2023-02-11 04:01:00,128 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK] 2023-02-11 04:01:00,130 DEBUG [jenkins-hbase4:45001] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:01:00,130 DEBUG [jenkins-hbase4:45001] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:01:00,130 DEBUG [jenkins-hbase4:45001] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:01:00,130 DEBUG [jenkins-hbase4:45001] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:01:00,132 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43137,1676088059590, state=OPENING 2023-02-11 04:01:00,133 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK] 2023-02-11 04:01:00,133 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK] 2023-02-11 04:01:00,133 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK] 2023-02-11 04:01:00,134 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 04:01:00,136 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:01:00,137 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 04:01:00,137 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43137,1676088059590}] 2023-02-11 04:01:00,140 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK] 2023-02-11 04:01:00,141 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK] 2023-02-11 04:01:00,140 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK] 2023-02-11 04:01:00,144 INFO [RS:2;jenkins-hbase4:36477] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36477,1676088059647/jenkins-hbase4.apache.org%2C36477%2C1676088059647.1676088060110 2023-02-11 04:01:00,147 DEBUG [RS:2;jenkins-hbase4:36477] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK], DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK], DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK]] 2023-02-11 04:01:00,148 INFO [RS:1;jenkins-hbase4:36163] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36163,1676088059618/jenkins-hbase4.apache.org%2C36163%2C1676088059618.1676088060111 2023-02-11 04:01:00,148 INFO [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,43137,1676088059590/jenkins-hbase4.apache.org%2C43137%2C1676088059590.1676088060107 2023-02-11 04:01:00,149 DEBUG [RS:1;jenkins-hbase4:36163] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK], DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK], DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK]] 2023-02-11 04:01:00,149 DEBUG [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK], DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK], DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK]] 2023-02-11 04:01:00,299 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:00,300 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:01:00,302 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54308, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:01:00,306 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 04:01:00,306 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:01:00,309 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43137%2C1676088059590.meta, suffix=.meta, logDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,43137,1676088059590, archiveDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs, maxLogs=32 2023-02-11 04:01:00,325 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK] 2023-02-11 04:01:00,325 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK] 2023-02-11 04:01:00,325 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK] 2023-02-11 04:01:00,328 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,43137,1676088059590/jenkins-hbase4.apache.org%2C43137%2C1676088059590.meta.1676088060309.meta 2023-02-11 04:01:00,328 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36549,DS-9060cc9f-1844-487f-9ba9-9cbaac906e6e,DISK], DatanodeInfoWithStorage[127.0.0.1:38521,DS-7058df01-f4b7-46c2-9caa-acc06dafaa5b,DISK], DatanodeInfoWithStorage[127.0.0.1:41997,DS-43a84765-a839-43d9-8bef-247d1fdacafc,DISK]] 2023-02-11 04:01:00,328 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:01:00,328 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 04:01:00,328 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 04:01:00,330 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 04:01:00,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 04:01:00,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:01:00,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 04:01:00,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 04:01:00,335 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 04:01:00,336 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/info 2023-02-11 04:01:00,336 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/info 2023-02-11 04:01:00,336 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 04:01:00,337 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:01:00,337 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 04:01:00,338 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:01:00,338 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:01:00,338 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 04:01:00,339 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:01:00,339 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 04:01:00,340 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/table 2023-02-11 04:01:00,340 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/table 2023-02-11 04:01:00,340 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 04:01:00,341 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:01:00,342 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740 2023-02-11 04:01:00,344 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740 2023-02-11 04:01:00,348 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 04:01:00,349 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 04:01:00,350 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70845457, jitterRate=0.055679574608802795}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 04:01:00,350 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676088060331Writing region info on filesystem at 1676088060331Initializing all the Stores at 1676088060332 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088060332Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088060334 (+2 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088060334Cleaning up temporary data from old regions at 1676088060345 (+11 ms)Cleaning up detritus from prior splits at 1676088060347 (+2 ms)Running coprocessor post-open hooks at 1676088060350 (+3 ms)Region opened successfully at 1676088060350 2023-02-11 04:01:00,352 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676088060299 2023-02-11 04:01:00,356 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 04:01:00,357 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 04:01:00,359 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43137,1676088059590, state=OPEN 2023-02-11 04:01:00,361 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 04:01:00,361 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 04:01:00,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 04:01:00,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43137,1676088059590 in 224 msec 2023-02-11 04:01:00,367 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 04:01:00,367 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 405 msec 2023-02-11 04:01:00,369 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 527 msec 2023-02-11 04:01:00,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676088059666, completionTime=-1 2023-02-11 04:01:00,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 04:01:00,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 04:01:00,372 DEBUG [hconnection-0x5d434a98-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:01:00,374 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54316, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:01:00,376 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 04:01:00,376 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676088120376 2023-02-11 04:01:00,376 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676088180376 2023-02-11 04:01:00,376 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 6 msec 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45001,1676088059528-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45001,1676088059528-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45001,1676088059528-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45001, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 04:01:00,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 04:01:00,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 04:01:00,384 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 04:01:00,386 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 04:01:00,387 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 04:01:00,389 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/.tmp/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,389 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/.tmp/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc empty. 2023-02-11 04:01:00,390 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/.tmp/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,390 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 04:01:00,405 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 04:01:00,407 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 6d28cb91b0071fbf1da05ec80fd7aecc, NAME => 'hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/.tmp 2023-02-11 04:01:00,422 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:01:00,422 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 6d28cb91b0071fbf1da05ec80fd7aecc, disabling compactions & flushes 2023-02-11 04:01:00,422 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,423 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,423 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. after waiting 0 ms 2023-02-11 04:01:00,423 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,423 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,423 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 6d28cb91b0071fbf1da05ec80fd7aecc: Waiting for close lock at 1676088060422Disabling compacts and flushes for region at 1676088060422Disabling writes for close at 1676088060423 (+1 ms)Writing region close event to WAL at 1676088060423Closed at 1676088060423 2023-02-11 04:01:00,426 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 04:01:00,428 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088060427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088060427"}]},"ts":"1676088060427"} 2023-02-11 04:01:00,430 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 04:01:00,432 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 04:01:00,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088060432"}]},"ts":"1676088060432"} 2023-02-11 04:01:00,434 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 04:01:00,442 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:01:00,443 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:01:00,443 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:01:00,443 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:01:00,443 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:01:00,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6d28cb91b0071fbf1da05ec80fd7aecc, ASSIGN}] 2023-02-11 04:01:00,446 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6d28cb91b0071fbf1da05ec80fd7aecc, ASSIGN 2023-02-11 04:01:00,447 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6d28cb91b0071fbf1da05ec80fd7aecc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36477,1676088059647; forceNewPlan=false, retain=false 2023-02-11 04:01:00,597 INFO [jenkins-hbase4:45001] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 04:01:00,598 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6d28cb91b0071fbf1da05ec80fd7aecc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:00,599 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088060598"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088060598"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088060598"}]},"ts":"1676088060598"} 2023-02-11 04:01:00,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 6d28cb91b0071fbf1da05ec80fd7aecc, server=jenkins-hbase4.apache.org,36477,1676088059647}] 2023-02-11 04:01:00,755 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:00,756 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:01:00,757 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56704, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:01:00,762 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,762 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d28cb91b0071fbf1da05ec80fd7aecc, NAME => 'hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:01:00,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:01:00,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,764 INFO [StoreOpener-6d28cb91b0071fbf1da05ec80fd7aecc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,766 DEBUG [StoreOpener-6d28cb91b0071fbf1da05ec80fd7aecc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/info 2023-02-11 04:01:00,766 DEBUG [StoreOpener-6d28cb91b0071fbf1da05ec80fd7aecc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/info 2023-02-11 04:01:00,767 INFO [StoreOpener-6d28cb91b0071fbf1da05ec80fd7aecc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d28cb91b0071fbf1da05ec80fd7aecc columnFamilyName info 2023-02-11 04:01:00,768 INFO [StoreOpener-6d28cb91b0071fbf1da05ec80fd7aecc-1] regionserver.HStore(310): Store=6d28cb91b0071fbf1da05ec80fd7aecc/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:01:00,769 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,770 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,773 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:00,777 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:01:00,778 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6d28cb91b0071fbf1da05ec80fd7aecc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64127086, jitterRate=-0.04443195462226868}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:01:00,778 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6d28cb91b0071fbf1da05ec80fd7aecc: Running coprocessor pre-open hook at 1676088060763Writing region info on filesystem at 1676088060763Initializing all the Stores at 1676088060764 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088060764Cleaning up temporary data from old regions at 1676088060771 (+7 ms)Cleaning up detritus from prior splits at 1676088060771Running coprocessor post-open hooks at 1676088060778 (+7 ms)Region opened successfully at 1676088060778 2023-02-11 04:01:00,779 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc., pid=6, masterSystemTime=1676088060755 2023-02-11 04:01:00,782 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,783 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:00,784 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6d28cb91b0071fbf1da05ec80fd7aecc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:00,784 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088060783"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088060783"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088060783"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088060783"}]},"ts":"1676088060783"} 2023-02-11 04:01:00,789 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 04:01:00,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 6d28cb91b0071fbf1da05ec80fd7aecc, server=jenkins-hbase4.apache.org,36477,1676088059647 in 185 msec 2023-02-11 04:01:00,793 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 04:01:00,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6d28cb91b0071fbf1da05ec80fd7aecc, ASSIGN in 346 msec 2023-02-11 04:01:00,795 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 04:01:00,795 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088060795"}]},"ts":"1676088060795"} 2023-02-11 04:01:00,797 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 04:01:00,799 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 04:01:00,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 416 msec 2023-02-11 04:01:00,885 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 04:01:00,887 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:01:00,887 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:01:00,890 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:01:00,892 INFO [RS-EventLoopGroup-10-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56718, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:01:00,894 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 04:01:00,907 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:01:00,911 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 16 msec 2023-02-11 04:01:00,917 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 04:01:00,925 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:01:00,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 11 msec 2023-02-11 04:01:00,942 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 04:01:00,944 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 04:01:00,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.272sec 2023-02-11 04:01:00,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 04:01:00,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 04:01:00,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 04:01:00,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45001,1676088059528-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 04:01:00,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45001,1676088059528-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 04:01:00,946 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 04:01:00,964 DEBUG [Listener at localhost/33533] zookeeper.ReadOnlyZKClient(139): Connect 0x36ae13be to 127.0.0.1:51435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:01:00,970 DEBUG [Listener at localhost/33533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a5d71cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:01:00,971 DEBUG [hconnection-0x46b97e73-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:01:00,974 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54324, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:01:00,975 INFO [Listener at localhost/33533] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:01:00,976 DEBUG [Listener at localhost/33533] zookeeper.ReadOnlyZKClient(139): Connect 0x775884c0 to 127.0.0.1:51435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:01:00,982 DEBUG [ReadOnlyZKClient-127.0.0.1:51435@0x775884c0] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4187bd8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:01:00,983 DEBUG [Listener at localhost/33533] client.ConnectionUtils(586): Start fetching master stub from registry 2023-02-11 04:01:00,984 DEBUG [ReadOnlyZKClient-127.0.0.1:51435@0x775884c0] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:01:00,984 DEBUG [ReadOnlyZKClient-127.0.0.1:51435@0x775884c0] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@27ef83 2023-02-11 04:01:00,987 DEBUG [ReadOnlyZKClient-127.0.0.1:51435@0x775884c0] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 04:01:00,989 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42842, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 04:01:00,989 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45001] master.MasterRpcServices(1560): Client=jenkins//172.31.14.131 shutdown 2023-02-11 04:01:00,989 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45001] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:01:00,991 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:01:00,991 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:01:00,991 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45001] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 04:01:00,991 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:01:00,991 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:01:00,992 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:01:00,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:01:00,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:01:00,993 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45001] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa7d30b to 127.0.0.1:51435 2023-02-11 04:01:00,993 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45001] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:00,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:01:00,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:01:01,063 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36163,1676088059618' ***** 2023-02-11 04:01:01,063 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-11 04:01:01,063 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1065): Closing user regions 2023-02-11 04:01:01,064 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(3304): Received CLOSE for 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:01,064 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1065): Closing user regions 2023-02-11 04:01:01,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6d28cb91b0071fbf1da05ec80fd7aecc, disabling compactions & flushes 2023-02-11 04:01:01,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:01,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:01,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. after waiting 0 ms 2023-02-11 04:01:01,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:01,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6d28cb91b0071fbf1da05ec80fd7aecc 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 04:01:01,065 INFO [RS:1;jenkins-hbase4:36163] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:01:01,066 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:01:01,066 INFO [RS:1;jenkins-hbase4:36163] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:01:01,066 INFO [RS:1;jenkins-hbase4:36163] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:01:01,066 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:01:01,066 DEBUG [RS:1;jenkins-hbase4:36163] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59a12e0a to 127.0.0.1:51435 2023-02-11 04:01:01,066 DEBUG [RS:1;jenkins-hbase4:36163] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,067 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36163,1676088059618; all regions closed. 2023-02-11 04:01:01,071 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/WALs/jenkins-hbase4.apache.org,36163,1676088059618/jenkins-hbase4.apache.org%2C36163%2C1676088059618.1676088060111 not finished, retry = 0 2023-02-11 04:01:01,074 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:01:01,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/.tmp/info/c8b78bdd2d09465aa6d9ef6072f5d731 2023-02-11 04:01:01,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/.tmp/info/c8b78bdd2d09465aa6d9ef6072f5d731 as hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/info/c8b78bdd2d09465aa6d9ef6072f5d731 2023-02-11 04:01:01,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/info/c8b78bdd2d09465aa6d9ef6072f5d731, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 04:01:01,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 6d28cb91b0071fbf1da05ec80fd7aecc in 48ms, sequenceid=6, compaction requested=false 2023-02-11 04:01:01,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/namespace/6d28cb91b0071fbf1da05ec80fd7aecc/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 04:01:01,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:01,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6d28cb91b0071fbf1da05ec80fd7aecc: Waiting for close lock at 1676088061064Running coprocessor pre-close hooks at 1676088061064Disabling compacts and flushes for region at 1676088061064Disabling writes for close at 1676088061065 (+1 ms)Obtaining lock to block concurrent updates at 1676088061065Preparing flush snapshotting stores in 6d28cb91b0071fbf1da05ec80fd7aecc at 1676088061065Finished memstore snapshotting hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676088061066 (+1 ms)Flushing stores of hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. at 1676088061067 (+1 ms)Flushing 6d28cb91b0071fbf1da05ec80fd7aecc/info: creating writer at 1676088061067Flushing 6d28cb91b0071fbf1da05ec80fd7aecc/info: appending metadata at 1676088061074 (+7 ms)Flushing 6d28cb91b0071fbf1da05ec80fd7aecc/info: closing flushed file at 1676088061074Flushing 6d28cb91b0071fbf1da05ec80fd7aecc/info: reopening flushed file at 1676088061105 (+31 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 6d28cb91b0071fbf1da05ec80fd7aecc in 48ms, sequenceid=6, compaction requested=false at 1676088061113 (+8 ms)Writing region close event to WAL at 1676088061115 (+2 ms)Running coprocessor post-close hooks at 1676088061121 (+6 ms)Closed at 1676088061121 2023-02-11 04:01:01,122 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45001] assignment.AssignmentManager(1094): RegionServer CLOSED 6d28cb91b0071fbf1da05ec80fd7aecc 2023-02-11 04:01:01,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676088060383.6d28cb91b0071fbf1da05ec80fd7aecc. 2023-02-11 04:01:01,167 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36477,1676088059647' ***** 2023-02-11 04:01:01,167 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-11 04:01:01,167 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1084): Waiting on 1588230740 2023-02-11 04:01:01,168 INFO [RS:2;jenkins-hbase4:36477] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:01:01,168 INFO [RS:2;jenkins-hbase4:36477] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:01:01,168 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:01:01,168 INFO [RS:2;jenkins-hbase4:36477] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:01:01,169 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:01,169 DEBUG [RS:2;jenkins-hbase4:36477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x015560e1 to 127.0.0.1:51435 2023-02-11 04:01:01,169 DEBUG [RS:2;jenkins-hbase4:36477] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,169 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36477,1676088059647; all regions closed. 2023-02-11 04:01:01,175 DEBUG [RS:2;jenkins-hbase4:36477] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs 2023-02-11 04:01:01,175 INFO [RS:2;jenkins-hbase4:36477] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36477%2C1676088059647:(num 1676088060110) 2023-02-11 04:01:01,175 DEBUG [RS:2;jenkins-hbase4:36477] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,175 INFO [RS:2;jenkins-hbase4:36477] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:01:01,175 DEBUG [RS:1;jenkins-hbase4:36163] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs 2023-02-11 04:01:01,175 INFO [RS:1;jenkins-hbase4:36163] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36163%2C1676088059618:(num 1676088060111) 2023-02-11 04:01:01,175 INFO [RS:2;jenkins-hbase4:36477] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 04:01:01,175 DEBUG [RS:1;jenkins-hbase4:36163] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,176 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:01:01,176 INFO [RS:1;jenkins-hbase4:36163] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:01:01,175 INFO [RS:2;jenkins-hbase4:36477] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:01:01,176 INFO [RS:2;jenkins-hbase4:36477] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:01:01,176 INFO [RS:2;jenkins-hbase4:36477] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:01:01,176 INFO [RS:1;jenkins-hbase4:36163] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 04:01:01,176 INFO [RS:1;jenkins-hbase4:36163] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:01:01,176 INFO [RS:2;jenkins-hbase4:36477] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36477 2023-02-11 04:01:01,176 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:01:01,177 INFO [RS:1;jenkins-hbase4:36163] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:01:01,177 INFO [RS:1;jenkins-hbase4:36163] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:01:01,179 INFO [RS:1;jenkins-hbase4:36163] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36163 2023-02-11 04:01:01,179 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:01:01,183 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:01:01,184 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:01:01,184 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:01:01,184 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c73bb37 rejected from java.util.concurrent.ThreadPoolExecutor@584c3461[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,184 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36477,1676088059647] 2023-02-11 04:01:01,184 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 2023-02-11 04:01:01,185 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36477,1676088059647; numProcessing=1 2023-02-11 04:01:01,185 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78c9d987 rejected from java.util.concurrent.ThreadPoolExecutor@7cf64246[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,187 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36477,1676088059647 already deleted, retry=false 2023-02-11 04:01:01,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:01,187 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36477,1676088059647 expired; onlineServers=2 2023-02-11 04:01:01,188 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36163,1676088059618] 2023-02-11 04:01:01,188 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,36477,1676088059647 znode expired, triggering replicatorRemoved event 2023-02-11 04:01:01,188 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36163,1676088059618; numProcessing=2 2023-02-11 04:01:01,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:01,189 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36163,1676088059618 already deleted, retry=false 2023-02-11 04:01:01,189 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36163,1676088059618 expired; onlineServers=1 2023-02-11 04:01:01,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:01,190 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,36163,1676088059618 znode expired, triggering replicatorRemoved event 2023-02-11 04:01:01,269 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43137,1676088059590' ***** 2023-02-11 04:01:01,269 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(2310): STOPPED: Stopped; only catalog regions remaining online 2023-02-11 04:01:01,270 INFO [RS:0;jenkins-hbase4:43137] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:01:01,270 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:01:01,270 INFO [RS:0;jenkins-hbase4:43137] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:01:01,270 INFO [RS:0;jenkins-hbase4:43137] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:01:01,271 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:01,271 DEBUG [RS:0;jenkins-hbase4:43137] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56861c35 to 127.0.0.1:51435 2023-02-11 04:01:01,271 DEBUG [RS:0;jenkins-hbase4:43137] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,271 INFO [RS:0;jenkins-hbase4:43137] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:01:01,271 INFO [RS:0;jenkins-hbase4:43137] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:01:01,271 INFO [RS:0;jenkins-hbase4:43137] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:01:01,271 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 04:01:01,271 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-11 04:01:01,271 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-11 04:01:01,272 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 04:01:01,272 DEBUG [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-11 04:01:01,272 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 04:01:01,272 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 04:01:01,272 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 04:01:01,272 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 04:01:01,272 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-02-11 04:01:01,275 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:01:01,288 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/.tmp/info/7b7930846405438b9eca64da5ceb5b62 2023-02-11 04:01:01,295 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,295 INFO [RS:1;jenkins-hbase4:36163] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36163,1676088059618; zookeeper connection closed. 2023-02-11 04:01:01,295 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3eb41a2d rejected from java.util.concurrent.ThreadPoolExecutor@7cf64246[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,295 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@e80e97] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@e80e97 2023-02-11 04:01:01,295 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x100418a102f0002, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,296 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@46ad7ea9 rejected from java.util.concurrent.ThreadPoolExecutor@7cf64246[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,310 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/.tmp/table/e5e7ed6232494daba75d82aa2b89dd71 2023-02-11 04:01:01,318 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/.tmp/info/7b7930846405438b9eca64da5ceb5b62 as hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/info/7b7930846405438b9eca64da5ceb5b62 2023-02-11 04:01:01,324 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/info/7b7930846405438b9eca64da5ceb5b62, entries=10, sequenceid=9, filesize=5.9 K 2023-02-11 04:01:01,326 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/.tmp/table/e5e7ed6232494daba75d82aa2b89dd71 as hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/table/e5e7ed6232494daba75d82aa2b89dd71 2023-02-11 04:01:01,332 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/table/e5e7ed6232494daba75d82aa2b89dd71, entries=2, sequenceid=9, filesize=4.7 K 2023-02-11 04:01:01,334 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 62ms, sequenceid=9, compaction requested=false 2023-02-11 04:01:01,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-02-11 04:01:01,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 04:01:01,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 04:01:01,342 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676088061271Running coprocessor pre-close hooks at 1676088061271Disabling compacts and flushes for region at 1676088061271Disabling writes for close at 1676088061272 (+1 ms)Obtaining lock to block concurrent updates at 1676088061272Preparing flush snapshotting stores in 1588230740 at 1676088061272Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1290, getHeapSize=2912, getOffHeapSize=0, getCellsCount=12 at 1676088061273 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676088061273Flushing 1588230740/info: creating writer at 1676088061273Flushing 1588230740/info: appending metadata at 1676088061277 (+4 ms)Flushing 1588230740/info: closing flushed file at 1676088061277Flushing 1588230740/table: creating writer at 1676088061297 (+20 ms)Flushing 1588230740/table: appending metadata at 1676088061300 (+3 ms)Flushing 1588230740/table: closing flushed file at 1676088061300Flushing 1588230740/info: reopening flushed file at 1676088061319 (+19 ms)Flushing 1588230740/table: reopening flushed file at 1676088061327 (+8 ms)Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 62ms, sequenceid=9, compaction requested=false at 1676088061334 (+7 ms)Writing region close event to WAL at 1676088061337 (+3 ms)Running coprocessor post-close hooks at 1676088061341 (+4 ms)Closed at 1676088061341 2023-02-11 04:01:01,342 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 04:01:01,395 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,395 INFO [RS:2;jenkins-hbase4:36477] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36477,1676088059647; zookeeper connection closed. 2023-02-11 04:01:01,396 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6d722963 rejected from java.util.concurrent.ThreadPoolExecutor@584c3461[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,396 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@602ac3ad] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@602ac3ad 2023-02-11 04:01:01,396 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:36477-0x100418a102f0003, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,396 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@15cf4b27 rejected from java.util.concurrent.ThreadPoolExecutor@584c3461[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,472 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43137,1676088059590; all regions closed. 2023-02-11 04:01:01,479 DEBUG [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs 2023-02-11 04:01:01,479 INFO [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43137%2C1676088059590.meta:.meta(num 1676088060309) 2023-02-11 04:01:01,485 DEBUG [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/oldWALs 2023-02-11 04:01:01,485 INFO [RS:0;jenkins-hbase4:43137] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43137%2C1676088059590:(num 1676088060107) 2023-02-11 04:01:01,485 DEBUG [RS:0;jenkins-hbase4:43137] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,485 INFO [RS:0;jenkins-hbase4:43137] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:01:01,485 INFO [RS:0;jenkins-hbase4:43137] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 04:01:01,485 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:01:01,486 INFO [RS:0;jenkins-hbase4:43137] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43137 2023-02-11 04:01:01,491 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:01:01,491 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43137,1676088059590 2023-02-11 04:01:01,491 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4d4205b2 rejected from java.util.concurrent.ThreadPoolExecutor@497d88c2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,491 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:01:01,491 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@600be5c7 rejected from java.util.concurrent.ThreadPoolExecutor@497d88c2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,493 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43137,1676088059590] 2023-02-11 04:01:01,493 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43137,1676088059590; numProcessing=3 2023-02-11 04:01:01,494 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43137,1676088059590 already deleted, retry=false 2023-02-11 04:01:01,494 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43137,1676088059590 expired; onlineServers=0 2023-02-11 04:01:01,494 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45001,1676088059528' ***** 2023-02-11 04:01:01,494 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-11 04:01:01,495 DEBUG [M:0;jenkins-hbase4:45001] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fff7bf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:01:01,495 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45001,1676088059528 2023-02-11 04:01:01,495 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45001,1676088059528; all regions closed. 2023-02-11 04:01:01,495 DEBUG [M:0;jenkins-hbase4:45001] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,495 DEBUG [M:0;jenkins-hbase4:45001] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 04:01:01,495 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 04:01:01,495 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088059853] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088059853,5,FailOnTimeoutGroup] 2023-02-11 04:01:01,495 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088059849] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088059849,5,FailOnTimeoutGroup] 2023-02-11 04:01:01,495 DEBUG [M:0;jenkins-hbase4:45001] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 04:01:01,497 INFO [M:0;jenkins-hbase4:45001] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 04:01:01,497 INFO [M:0;jenkins-hbase4:45001] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 04:01:01,497 INFO [M:0;jenkins-hbase4:45001] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 04:01:01,497 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 04:01:01,497 DEBUG [M:0;jenkins-hbase4:45001] master.HMaster(1502): Stopping service threads 2023-02-11 04:01:01,497 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:01:01,497 INFO [M:0;jenkins-hbase4:45001] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 04:01:01,497 INFO [Listener at localhost/33533] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/33533. 2023-02-11 04:01:01,498 ERROR [M:0;jenkins-hbase4:45001] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-2,5,PEWorkerGroup] 2023-02-11 04:01:01,498 DEBUG [Listener at localhost/33533] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2.tearDown(TestAsyncClusterAdminApi2.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:01:01,498 DEBUG [Listener at localhost/33533] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,498 INFO [M:0;jenkins-hbase4:45001] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 04:01:01,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:01:01,499 DEBUG [Listener at localhost/33533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x775884c0 to 127.0.0.1:51435 2023-02-11 04:01:01,499 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 04:01:01,499 INFO [Listener at localhost/33533] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 04:01:01,499 DEBUG [M:0;jenkins-hbase4:45001] zookeeper.ZKUtil(398): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 04:01:01,499 DEBUG [Listener at localhost/33533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36ae13be to 127.0.0.1:51435 2023-02-11 04:01:01,499 WARN [M:0;jenkins-hbase4:45001] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 04:01:01,499 DEBUG [Listener at localhost/33533] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:01:01,499 INFO [M:0;jenkins-hbase4:45001] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 04:01:01,500 DEBUG [Listener at localhost/33533] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 04:01:01,500 INFO [M:0;jenkins-hbase4:45001] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 04:01:01,501 DEBUG [M:0;jenkins-hbase4:45001] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 04:01:01,501 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:01:01,501 DEBUG [M:0;jenkins-hbase4:45001] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:01:01,501 DEBUG [M:0;jenkins-hbase4:45001] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 04:01:01,501 DEBUG [M:0;jenkins-hbase4:45001] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:01:01,501 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.07 KB heapSize=29.55 KB 2023-02-11 04:01:01,518 INFO [M:0;jenkins-hbase4:45001] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e99cda4a7742464ca28381c82014eb1d 2023-02-11 04:01:01,525 DEBUG [M:0;jenkins-hbase4:45001] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e99cda4a7742464ca28381c82014eb1d as hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e99cda4a7742464ca28381c82014eb1d 2023-02-11 04:01:01,532 INFO [M:0;jenkins-hbase4:45001] regionserver.HStore(1080): Added hdfs://localhost:41045/user/jenkins/test-data/c4d6069b-2331-840d-e7e3-0bf6ec7b29c1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e99cda4a7742464ca28381c82014eb1d, entries=8, sequenceid=66, filesize=6.3 K 2023-02-11 04:01:01,533 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegion(2947): Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=66, compaction requested=false 2023-02-11 04:01:01,534 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:01:01,534 DEBUG [M:0;jenkins-hbase4:45001] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676088061501Disabling compacts and flushes for region at 1676088061501Disabling writes for close at 1676088061501Obtaining lock to block concurrent updates at 1676088061501Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676088061501Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=24646, getHeapSize=30248, getOffHeapSize=0, getCellsCount=71 at 1676088061502 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676088061503 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676088061503Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676088061507 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676088061507Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676088061526 (+19 ms)Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=66, compaction requested=false at 1676088061533 (+7 ms)Writing region close event to WAL at 1676088061534 (+1 ms)Closed at 1676088061534 2023-02-11 04:01:01,538 INFO [M:0;jenkins-hbase4:45001] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 04:01:01,538 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:01:01,540 INFO [M:0;jenkins-hbase4:45001] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45001 2023-02-11 04:01:01,541 DEBUG [M:0;jenkins-hbase4:45001] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45001,1676088059528 already deleted, retry=false 2023-02-11 04:01:01,596 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-11 04:01:01,596 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,596 INFO [RS:0;jenkins-hbase4:43137] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43137,1676088059590; zookeeper connection closed. 2023-02-11 04:01:01,596 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@18a7b71c rejected from java.util.concurrent.ThreadPoolExecutor@497d88c2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,596 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@61ddd069] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@61ddd069 2023-02-11 04:01:01,597 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): regionserver:43137-0x100418a102f0001, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,597 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@53ca1556 rejected from java.util.concurrent.ThreadPoolExecutor@497d88c2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,597 INFO [Listener at localhost/33533] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 04:01:01,797 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,797 INFO [M:0;jenkins-hbase4:45001] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45001,1676088059528; zookeeper connection closed. 2023-02-11 04:01:01,797 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7bad4f24 rejected from java.util.concurrent.ThreadPoolExecutor@7f92d489[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,798 DEBUG [Listener at localhost/33533-EventThread] zookeeper.ZKWatcher(600): master:45001-0x100418a102f0000, quorum=127.0.0.1:51435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:01:01,798 ERROR [Listener at localhost/33533-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1a25b0c3 rejected from java.util.concurrent.ThreadPoolExecutor@7f92d489[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:01:01,798 WARN [Listener at localhost/33533] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:01:01,802 INFO [Listener at localhost/33533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:01:01,905 WARN [BP-1375890542-172.31.14.131-1676088058677 heartbeating to localhost/127.0.0.1:41045] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:01:01,905 WARN [BP-1375890542-172.31.14.131-1676088058677 heartbeating to localhost/127.0.0.1:41045] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1375890542-172.31.14.131-1676088058677 (Datanode Uuid 2124e074-f7f0-440b-b0fb-3bf35e8e3e5e) service to localhost/127.0.0.1:41045 2023-02-11 04:01:01,906 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/dfs/data/data5/current/BP-1375890542-172.31.14.131-1676088058677] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:01:01,906 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/dfs/data/data6/current/BP-1375890542-172.31.14.131-1676088058677] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:01:01,907 WARN [Listener at localhost/33533] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:01:01,910 INFO [Listener at localhost/33533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:01:02,015 WARN [BP-1375890542-172.31.14.131-1676088058677 heartbeating to localhost/127.0.0.1:41045] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:01:02,015 WARN [BP-1375890542-172.31.14.131-1676088058677 heartbeating to localhost/127.0.0.1:41045] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1375890542-172.31.14.131-1676088058677 (Datanode Uuid 042d5c25-1621-492b-8e44-10f3ca3f3485) service to localhost/127.0.0.1:41045 2023-02-11 04:01:02,016 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/dfs/data/data3/current/BP-1375890542-172.31.14.131-1676088058677] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:01:02,016 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/dfs/data/data4/current/BP-1375890542-172.31.14.131-1676088058677] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:01:02,017 WARN [Listener at localhost/33533] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:01:02,022 INFO [Listener at localhost/33533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:01:02,097 WARN [BP-1375890542-172.31.14.131-1676088058677 heartbeating to localhost/127.0.0.1:41045] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1375890542-172.31.14.131-1676088058677 (Datanode Uuid ba3e46fc-0706-4b4a-ad86-8f7bbbeb9807) service to localhost/127.0.0.1:41045 2023-02-11 04:01:02,098 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/dfs/data/data1/current/BP-1375890542-172.31.14.131-1676088058677] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:01:02,098 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0566c872-dd62-4bae-a948-556e214a765d/cluster_97625b0a-fb5a-2716-fa46-a18e42f20fc6/dfs/data/data2/current/BP-1375890542-172.31.14.131-1676088058677] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:01:02,138 INFO [Listener at localhost/33533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:01:02,253 INFO [Listener at localhost/33533] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 04:01:02,285 INFO [Listener at localhost/33533] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-11 04:01:02,296 INFO [Listener at localhost/33533] hbase.ResourceChecker(175): after: client.TestAsyncClusterAdminApi2#testShutdown Thread=108 (was 79) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:41045 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:41045 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:41045 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:41045 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/33533 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:41045 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:41045 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:41045 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Client (1523191903) connection to localhost/127.0.0.1:41045 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:41045 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=543 (was 504) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=248 (was 261), ProcessCount=169 (was 169), AvailableMemoryMB=3062 (was 3146)