2023-02-09 18:00:18,397 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c 2023-02-09 18:00:18,412 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2 timeout: 13 mins 2023-02-09 18:00:18,448 INFO [Time-limited test] hbase.ResourceChecker(147): before: client.TestAsyncClusterAdminApi2#testStop Thread=8, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=167, AvailableMemoryMB=3877 2023-02-09 18:00:18,455 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-09 18:00:18,455 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886, deleteOnExit=true 2023-02-09 18:00:18,455 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-09 18:00:18,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/test.cache.data in system properties and HBase conf 2023-02-09 18:00:18,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/hadoop.tmp.dir in system properties and HBase conf 2023-02-09 18:00:18,457 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/hadoop.log.dir in system properties and HBase conf 2023-02-09 18:00:18,457 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-09 18:00:18,458 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-09 18:00:18,458 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-09 18:00:18,581 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-09 18:00:18,963 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-09 18:00:18,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-09 18:00:18,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-09 18:00:18,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-09 18:00:18,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-09 18:00:18,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-09 18:00:18,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-09 18:00:18,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-09 18:00:18,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-09 18:00:18,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-09 18:00:18,972 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/nfs.dump.dir in system properties and HBase conf 2023-02-09 18:00:18,972 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/java.io.tmpdir in system properties and HBase conf 2023-02-09 18:00:18,973 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-09 18:00:18,973 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-09 18:00:18,973 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-09 18:00:19,454 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-09 18:00:19,458 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-09 18:00:19,730 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-09 18:00:19,878 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-09 18:00:19,893 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:19,930 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:19,962 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/java.io.tmpdir/Jetty_localhost_36499_hdfs____7vzqz/webapp 2023-02-09 18:00:20,115 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36499 2023-02-09 18:00:20,126 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-09 18:00:20,126 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-09 18:00:20,552 WARN [Listener at localhost/44505] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:20,623 WARN [Listener at localhost/44505] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 18:00:20,641 WARN [Listener at localhost/44505] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:20,649 INFO [Listener at localhost/44505] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:20,654 INFO [Listener at localhost/44505] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/java.io.tmpdir/Jetty_localhost_41999_datanode____.vw9r4t/webapp 2023-02-09 18:00:20,750 INFO [Listener at localhost/44505] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41999 2023-02-09 18:00:21,071 WARN [Listener at localhost/37753] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:21,082 WARN [Listener at localhost/37753] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 18:00:21,085 WARN [Listener at localhost/37753] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:21,086 INFO [Listener at localhost/37753] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:21,092 INFO [Listener at localhost/37753] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/java.io.tmpdir/Jetty_localhost_34403_datanode____.svgyqr/webapp 2023-02-09 18:00:21,197 INFO [Listener at localhost/37753] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34403 2023-02-09 18:00:21,207 WARN [Listener at localhost/39611] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:21,222 WARN [Listener at localhost/39611] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 18:00:21,228 WARN [Listener at localhost/39611] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:21,229 INFO [Listener at localhost/39611] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:21,235 INFO [Listener at localhost/39611] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/java.io.tmpdir/Jetty_localhost_45435_datanode____lb1rz0/webapp 2023-02-09 18:00:21,336 INFO [Listener at localhost/39611] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45435 2023-02-09 18:00:21,344 WARN [Listener at localhost/38227] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:21,585 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x76626888c25c40c5: Processing first storage report for DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6 from datanode 6dd596e7-dc0f-4c35-80ba-5c41aa63abea 2023-02-09 18:00:21,587 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x76626888c25c40c5: from storage DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6 node DatanodeRegistration(127.0.0.1:43749, datanodeUuid=6dd596e7-dc0f-4c35-80ba-5c41aa63abea, infoPort=38781, infoSecurePort=0, ipcPort=39611, storageInfo=lv=-57;cid=testClusterID;nsid=380427520;c=1675965619531), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-09 18:00:21,587 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb756e05122c6e3fa: Processing first storage report for DS-a0cf1878-0ebd-4479-9583-f66a68830d81 from datanode 82264b30-6c2f-4134-9b35-60a3df5f7b06 2023-02-09 18:00:21,587 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb756e05122c6e3fa: from storage DS-a0cf1878-0ebd-4479-9583-f66a68830d81 node DatanodeRegistration(127.0.0.1:44441, datanodeUuid=82264b30-6c2f-4134-9b35-60a3df5f7b06, infoPort=44093, infoSecurePort=0, ipcPort=37753, storageInfo=lv=-57;cid=testClusterID;nsid=380427520;c=1675965619531), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:21,587 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x94d16a8f7f12045d: Processing first storage report for DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1 from datanode c9d26d5c-ecc3-44b5-b864-eece467a5b9b 2023-02-09 18:00:21,587 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x94d16a8f7f12045d: from storage DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1 node DatanodeRegistration(127.0.0.1:45057, datanodeUuid=c9d26d5c-ecc3-44b5-b864-eece467a5b9b, infoPort=33093, infoSecurePort=0, ipcPort=38227, storageInfo=lv=-57;cid=testClusterID;nsid=380427520;c=1675965619531), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:21,587 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x76626888c25c40c5: Processing first storage report for DS-b38386aa-74f2-4413-8d02-df56cc1daf1b from datanode 6dd596e7-dc0f-4c35-80ba-5c41aa63abea 2023-02-09 18:00:21,588 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x76626888c25c40c5: from storage DS-b38386aa-74f2-4413-8d02-df56cc1daf1b node DatanodeRegistration(127.0.0.1:43749, datanodeUuid=6dd596e7-dc0f-4c35-80ba-5c41aa63abea, infoPort=38781, infoSecurePort=0, ipcPort=39611, storageInfo=lv=-57;cid=testClusterID;nsid=380427520;c=1675965619531), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-09 18:00:21,588 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb756e05122c6e3fa: Processing first storage report for DS-01c5e207-eea7-4c93-8eb7-682bff79c94c from datanode 82264b30-6c2f-4134-9b35-60a3df5f7b06 2023-02-09 18:00:21,588 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb756e05122c6e3fa: from storage DS-01c5e207-eea7-4c93-8eb7-682bff79c94c node DatanodeRegistration(127.0.0.1:44441, datanodeUuid=82264b30-6c2f-4134-9b35-60a3df5f7b06, infoPort=44093, infoSecurePort=0, ipcPort=37753, storageInfo=lv=-57;cid=testClusterID;nsid=380427520;c=1675965619531), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:21,588 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x94d16a8f7f12045d: Processing first storage report for DS-b759b6ab-ca57-4f18-a58d-a5712912043a from datanode c9d26d5c-ecc3-44b5-b864-eece467a5b9b 2023-02-09 18:00:21,588 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x94d16a8f7f12045d: from storage DS-b759b6ab-ca57-4f18-a58d-a5712912043a node DatanodeRegistration(127.0.0.1:45057, datanodeUuid=c9d26d5c-ecc3-44b5-b864-eece467a5b9b, infoPort=33093, infoSecurePort=0, ipcPort=38227, storageInfo=lv=-57;cid=testClusterID;nsid=380427520;c=1675965619531), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:21,743 DEBUG [Listener at localhost/38227] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c 2023-02-09 18:00:21,806 INFO [Listener at localhost/38227] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/zookeeper_0, clientPort=49196, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-09 18:00:21,823 INFO [Listener at localhost/38227] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=49196 2023-02-09 18:00:21,830 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:21,832 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:22,501 INFO [Listener at localhost/38227] util.FSUtils(479): Created version file at hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6 with version=8 2023-02-09 18:00:22,501 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/hbase-staging 2023-02-09 18:00:22,807 INFO [Listener at localhost/38227] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-09 18:00:23,261 INFO [Listener at localhost/38227] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:23,294 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,295 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,295 INFO [Listener at localhost/38227] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:23,295 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,295 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:23,439 INFO [Listener at localhost/38227] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:23,512 DEBUG [Listener at localhost/38227] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-09 18:00:23,607 INFO [Listener at localhost/38227] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46015 2023-02-09 18:00:23,617 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,619 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,640 INFO [Listener at localhost/38227] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46015 connecting to ZooKeeper ensemble=127.0.0.1:49196 2023-02-09 18:00:23,684 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:460150x0, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:23,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46015-0x1003a3dcfe40000 connected 2023-02-09 18:00:23,710 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:23,711 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:23,715 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:23,723 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46015 2023-02-09 18:00:23,723 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46015 2023-02-09 18:00:23,723 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46015 2023-02-09 18:00:23,724 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46015 2023-02-09 18:00:23,724 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46015 2023-02-09 18:00:23,730 INFO [Listener at localhost/38227] master.HMaster(439): hbase.rootdir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6, hbase.cluster.distributed=false 2023-02-09 18:00:23,806 INFO [Listener at localhost/38227] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:23,807 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,807 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,807 INFO [Listener at localhost/38227] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:23,808 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,808 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:23,813 INFO [Listener at localhost/38227] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:23,817 INFO [Listener at localhost/38227] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36993 2023-02-09 18:00:23,819 INFO [Listener at localhost/38227] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 18:00:23,827 DEBUG [Listener at localhost/38227] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 18:00:23,829 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,831 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,833 INFO [Listener at localhost/38227] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36993 connecting to ZooKeeper ensemble=127.0.0.1:49196 2023-02-09 18:00:23,838 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:369930x0, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:23,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36993-0x1003a3dcfe40001 connected 2023-02-09 18:00:23,839 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:23,840 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:23,841 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:23,842 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36993 2023-02-09 18:00:23,842 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36993 2023-02-09 18:00:23,842 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36993 2023-02-09 18:00:23,844 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36993 2023-02-09 18:00:23,844 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36993 2023-02-09 18:00:23,857 INFO [Listener at localhost/38227] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:23,857 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,858 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,858 INFO [Listener at localhost/38227] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:23,858 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,859 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:23,859 INFO [Listener at localhost/38227] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:23,860 INFO [Listener at localhost/38227] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34757 2023-02-09 18:00:23,861 INFO [Listener at localhost/38227] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 18:00:23,863 DEBUG [Listener at localhost/38227] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 18:00:23,863 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,866 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,868 INFO [Listener at localhost/38227] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34757 connecting to ZooKeeper ensemble=127.0.0.1:49196 2023-02-09 18:00:23,874 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:347570x0, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:23,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34757-0x1003a3dcfe40002 connected 2023-02-09 18:00:23,875 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:23,876 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:23,877 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:23,878 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34757 2023-02-09 18:00:23,878 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34757 2023-02-09 18:00:23,879 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34757 2023-02-09 18:00:23,879 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34757 2023-02-09 18:00:23,879 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34757 2023-02-09 18:00:23,893 INFO [Listener at localhost/38227] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:23,893 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,893 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,893 INFO [Listener at localhost/38227] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:23,893 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:23,894 INFO [Listener at localhost/38227] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:23,894 INFO [Listener at localhost/38227] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:23,895 INFO [Listener at localhost/38227] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34863 2023-02-09 18:00:23,896 INFO [Listener at localhost/38227] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 18:00:23,897 DEBUG [Listener at localhost/38227] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 18:00:23,898 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,900 INFO [Listener at localhost/38227] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:23,902 INFO [Listener at localhost/38227] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34863 connecting to ZooKeeper ensemble=127.0.0.1:49196 2023-02-09 18:00:23,907 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:348630x0, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:23,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34863-0x1003a3dcfe40003 connected 2023-02-09 18:00:23,908 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:23,909 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:23,909 DEBUG [Listener at localhost/38227] zookeeper.ZKUtil(164): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:23,910 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34863 2023-02-09 18:00:23,910 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34863 2023-02-09 18:00:23,910 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34863 2023-02-09 18:00:23,911 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34863 2023-02-09 18:00:23,911 DEBUG [Listener at localhost/38227] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34863 2023-02-09 18:00:23,913 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:23,922 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-09 18:00:23,923 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:23,942 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:23,942 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:23,942 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:23,942 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:23,943 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:23,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-09 18:00:23,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46015,1675965622645 from backup master directory 2023-02-09 18:00:23,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-09 18:00:23,950 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:23,950 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-09 18:00:23,951 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:23,951 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:23,954 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-09 18:00:23,955 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-09 18:00:24,056 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/hbase.id with ID: 82b5cce9-02f4-41d7-b840-bca3063c981d 2023-02-09 18:00:24,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:24,116 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:24,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2d0608b6 to 127.0.0.1:49196 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:24,203 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57cd6a8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:24,228 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-09 18:00:24,230 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-09 18:00:24,248 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-09 18:00:24,248 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-09 18:00:24,251 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-09 18:00:24,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-09 18:00:24,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:24,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store-tmp 2023-02-09 18:00:24,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:24,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-09 18:00:24,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:24,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:24,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-09 18:00:24,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:24,361 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:24,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1675965624360Disabling compacts and flushes for region at 1675965624360Disabling writes for close at 1675965624361 (+1 ms)Writing region close event to WAL at 1675965624361Closed at 1675965624361 2023-02-09 18:00:24,363 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/WALs/jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:24,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46015%2C1675965622645, suffix=, logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/WALs/jenkins-hbase4.apache.org,46015,1675965622645, archiveDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/oldWALs, maxLogs=10 2023-02-09 18:00:24,447 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK] 2023-02-09 18:00:24,447 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK] 2023-02-09 18:00:24,447 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK] 2023-02-09 18:00:24,456 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-09 18:00:24,532 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/WALs/jenkins-hbase4.apache.org,46015,1675965622645/jenkins-hbase4.apache.org%2C46015%2C1675965622645.1675965624393 2023-02-09 18:00:24,533 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK], DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK], DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK]] 2023-02-09 18:00:24,533 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-09 18:00:24,534 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:24,538 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:24,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:24,597 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:24,604 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-09 18:00:24,628 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-09 18:00:24,639 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:24,646 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:24,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:24,663 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:24,667 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 18:00:24,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72374081, jitterRate=0.07845784723758698}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 18:00:24,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1675965624565Initializing all the Stores at 1675965624568 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675965624569 (+1 ms)Cleaning up temporary data from old regions at 1675965624654 (+85 ms)Cleaning up detritus from prior splits at 1675965624655 (+1 ms)Region opened successfully at 1675965624668 (+13 ms) 2023-02-09 18:00:24,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-09 18:00:24,688 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-09 18:00:24,688 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-09 18:00:24,690 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 18:00:24,692 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-09 18:00:24,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-09 18:00:24,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-09 18:00:24,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-09 18:00:24,754 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-09 18:00:24,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-09 18:00:24,781 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-09 18:00:24,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-09 18:00:24,787 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-09 18:00:24,791 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-09 18:00:24,794 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:24,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-09 18:00:24,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-09 18:00:24,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-09 18:00:24,817 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:24,817 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:24,817 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:24,817 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:24,817 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:24,818 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,46015,1675965622645, sessionid=0x1003a3dcfe40000, setting cluster-up flag (Was=false) 2023-02-09 18:00:24,833 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:24,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-09 18:00:24,841 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:24,845 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:24,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-09 18:00:24,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:24,855 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/.hbase-snapshot/.tmp 2023-02-09 18:00:24,915 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(952): ClusterId : 82b5cce9-02f4-41d7-b840-bca3063c981d 2023-02-09 18:00:24,915 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(952): ClusterId : 82b5cce9-02f4-41d7-b840-bca3063c981d 2023-02-09 18:00:24,915 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(952): ClusterId : 82b5cce9-02f4-41d7-b840-bca3063c981d 2023-02-09 18:00:24,921 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 18:00:24,921 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 18:00:24,921 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 18:00:24,929 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 18:00:24,929 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 18:00:24,929 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 18:00:24,929 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 18:00:24,932 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 18:00:24,933 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 18:00:24,933 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 18:00:24,933 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 18:00:24,935 DEBUG [RS:2;jenkins-hbase4:34863] zookeeper.ReadOnlyZKClient(139): Connect 0x54d0eb27 to 127.0.0.1:49196 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:24,935 DEBUG [RS:1;jenkins-hbase4:34757] zookeeper.ReadOnlyZKClient(139): Connect 0x3c3b4380 to 127.0.0.1:49196 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:24,936 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 18:00:24,939 DEBUG [RS:0;jenkins-hbase4:36993] zookeeper.ReadOnlyZKClient(139): Connect 0x79795eee to 127.0.0.1:49196 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:24,946 DEBUG [RS:1;jenkins-hbase4:34757] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32fcaa77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:24,946 DEBUG [RS:1;jenkins-hbase4:34757] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36c328f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:24,946 DEBUG [RS:2;jenkins-hbase4:34863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a171fcd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:24,947 DEBUG [RS:2;jenkins-hbase4:34863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e288057, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:24,945 DEBUG [RS:0;jenkins-hbase4:36993] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77beafb1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:24,947 DEBUG [RS:0;jenkins-hbase4:36993] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7381359e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:24,973 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:36993 2023-02-09 18:00:24,974 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:34757 2023-02-09 18:00:24,975 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:34863 2023-02-09 18:00:24,976 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-09 18:00:24,978 INFO [RS:2;jenkins-hbase4:34863] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 18:00:24,978 INFO [RS:1;jenkins-hbase4:34757] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 18:00:24,979 INFO [RS:2;jenkins-hbase4:34863] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 18:00:24,978 INFO [RS:0;jenkins-hbase4:36993] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 18:00:24,979 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 18:00:24,979 INFO [RS:1;jenkins-hbase4:34757] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 18:00:24,979 INFO [RS:0;jenkins-hbase4:36993] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 18:00:24,979 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 18:00:24,979 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 18:00:24,982 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46015,1675965622645 with isa=jenkins-hbase4.apache.org/172.31.14.131:34863, startcode=1675965623892 2023-02-09 18:00:24,982 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46015,1675965622645 with isa=jenkins-hbase4.apache.org/172.31.14.131:34757, startcode=1675965623856 2023-02-09 18:00:24,982 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46015,1675965622645 with isa=jenkins-hbase4.apache.org/172.31.14.131:36993, startcode=1675965623805 2023-02-09 18:00:24,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:24,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:24,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:24,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:24,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-09 18:00:24,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:24,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:24,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:24,989 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675965654989 2023-02-09 18:00:24,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-09 18:00:24,996 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-09 18:00:24,997 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-09 18:00:25,002 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-09 18:00:25,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-09 18:00:25,002 DEBUG [RS:2;jenkins-hbase4:34863] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 18:00:25,002 DEBUG [RS:0;jenkins-hbase4:36993] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 18:00:25,002 DEBUG [RS:1;jenkins-hbase4:34757] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 18:00:25,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-09 18:00:25,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-09 18:00:25,011 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-09 18:00:25,011 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-09 18:00:25,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-09 18:00:25,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-09 18:00:25,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-09 18:00:25,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-09 18:00:25,020 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-09 18:00:25,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1675965625021,5,FailOnTimeoutGroup] 2023-02-09 18:00:25,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1675965625022,5,FailOnTimeoutGroup] 2023-02-09 18:00:25,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-09 18:00:25,026 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,026 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,061 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-09 18:00:25,062 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-09 18:00:25,062 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6 2023-02-09 18:00:25,076 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39609, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 18:00:25,076 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57907, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 18:00:25,076 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48203, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 18:00:25,091 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:25,093 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-09 18:00:25,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/info 2023-02-09 18:00:25,097 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46015] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,097 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46015] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,098 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-09 18:00:25,099 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:25,099 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-09 18:00:25,102 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/rep_barrier 2023-02-09 18:00:25,103 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-09 18:00:25,104 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:25,104 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-09 18:00:25,106 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/table 2023-02-09 18:00:25,107 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-09 18:00:25,108 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:25,111 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740 2023-02-09 18:00:25,113 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740 2023-02-09 18:00:25,116 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6 2023-02-09 18:00:25,116 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6 2023-02-09 18:00:25,116 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6 2023-02-09 18:00:25,116 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44505 2023-02-09 18:00:25,116 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44505 2023-02-09 18:00:25,116 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 18:00:25,116 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44505 2023-02-09 18:00:25,117 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 18:00:25,117 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 18:00:25,118 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-09 18:00:25,120 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-09 18:00:25,125 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:25,126 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 18:00:25,127 DEBUG [RS:0;jenkins-hbase4:36993] zookeeper.ZKUtil(162): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,127 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68246758, jitterRate=0.016955941915512085}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-09 18:00:25,127 WARN [RS:0;jenkins-hbase4:36993] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:25,127 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1675965625091Initializing all the Stores at 1675965625093 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965625093Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675965625093Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965625093Cleaning up temporary data from old regions at 1675965625115 (+22 ms)Cleaning up detritus from prior splits at 1675965625116 (+1 ms)Region opened successfully at 1675965625127 (+11 ms) 2023-02-09 18:00:25,128 INFO [RS:0;jenkins-hbase4:36993] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:25,127 DEBUG [RS:1;jenkins-hbase4:34757] zookeeper.ZKUtil(162): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,128 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,128 WARN [RS:1;jenkins-hbase4:34757] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:25,128 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36993,1675965623805] 2023-02-09 18:00:25,129 DEBUG [RS:2;jenkins-hbase4:34863] zookeeper.ZKUtil(162): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,128 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-09 18:00:25,129 WARN [RS:2;jenkins-hbase4:34863] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:25,129 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34757,1675965623856] 2023-02-09 18:00:25,129 INFO [RS:1;jenkins-hbase4:34757] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:25,130 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34863,1675965623892] 2023-02-09 18:00:25,130 INFO [RS:2;jenkins-hbase4:34863] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:25,131 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,130 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-09 18:00:25,130 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,131 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-09 18:00:25,131 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-09 18:00:25,131 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-09 18:00:25,133 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-09 18:00:25,133 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1675965625128Disabling compacts and flushes for region at 1675965625128Disabling writes for close at 1675965625131 (+3 ms)Writing region close event to WAL at 1675965625133 (+2 ms)Closed at 1675965625133 2023-02-09 18:00:25,139 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-09 18:00:25,139 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-09 18:00:25,150 DEBUG [RS:1;jenkins-hbase4:34757] zookeeper.ZKUtil(162): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,150 DEBUG [RS:0;jenkins-hbase4:36993] zookeeper.ZKUtil(162): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,150 DEBUG [RS:2;jenkins-hbase4:34863] zookeeper.ZKUtil(162): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,151 DEBUG [RS:1;jenkins-hbase4:34757] zookeeper.ZKUtil(162): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,151 DEBUG [RS:0;jenkins-hbase4:36993] zookeeper.ZKUtil(162): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,151 DEBUG [RS:2;jenkins-hbase4:34863] zookeeper.ZKUtil(162): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,151 DEBUG [RS:1;jenkins-hbase4:34757] zookeeper.ZKUtil(162): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,152 DEBUG [RS:2;jenkins-hbase4:34863] zookeeper.ZKUtil(162): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,151 DEBUG [RS:0;jenkins-hbase4:36993] zookeeper.ZKUtil(162): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-09 18:00:25,162 DEBUG [RS:1;jenkins-hbase4:34757] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 18:00:25,162 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 18:00:25,162 DEBUG [RS:2;jenkins-hbase4:34863] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 18:00:25,166 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-09 18:00:25,170 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-09 18:00:25,173 INFO [RS:2;jenkins-hbase4:34863] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 18:00:25,174 INFO [RS:1;jenkins-hbase4:34757] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 18:00:25,173 INFO [RS:0;jenkins-hbase4:36993] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 18:00:25,195 INFO [RS:1;jenkins-hbase4:34757] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 18:00:25,195 INFO [RS:2;jenkins-hbase4:34863] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 18:00:25,195 INFO [RS:0;jenkins-hbase4:36993] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 18:00:25,199 INFO [RS:0;jenkins-hbase4:36993] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 18:00:25,199 INFO [RS:1;jenkins-hbase4:34757] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 18:00:25,199 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,199 INFO [RS:2;jenkins-hbase4:34863] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 18:00:25,199 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,200 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,200 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 18:00:25,201 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 18:00:25,201 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 18:00:25,209 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,209 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,209 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,209 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,209 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,209 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,209 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:25,210 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:25,210 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,210 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:25,210 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:0;jenkins-hbase4:36993] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:1;jenkins-hbase4:34757] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,211 DEBUG [RS:2;jenkins-hbase4:34863] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:25,213 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,213 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,214 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,229 INFO [RS:1;jenkins-hbase4:34757] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 18:00:25,229 INFO [RS:2;jenkins-hbase4:34863] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 18:00:25,232 INFO [RS:0;jenkins-hbase4:36993] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 18:00:25,232 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36993,1675965623805-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,232 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34757,1675965623856-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,232 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34863,1675965623892-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,252 INFO [RS:1;jenkins-hbase4:34757] regionserver.Replication(203): jenkins-hbase4.apache.org,34757,1675965623856 started 2023-02-09 18:00:25,252 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34757,1675965623856, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34757, sessionid=0x1003a3dcfe40002 2023-02-09 18:00:25,252 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 18:00:25,252 DEBUG [RS:1;jenkins-hbase4:34757] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,252 DEBUG [RS:1;jenkins-hbase4:34757] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34757,1675965623856' 2023-02-09 18:00:25,252 DEBUG [RS:1;jenkins-hbase4:34757] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 18:00:25,253 INFO [RS:2;jenkins-hbase4:34863] regionserver.Replication(203): jenkins-hbase4.apache.org,34863,1675965623892 started 2023-02-09 18:00:25,253 INFO [RS:0;jenkins-hbase4:36993] regionserver.Replication(203): jenkins-hbase4.apache.org,36993,1675965623805 started 2023-02-09 18:00:25,253 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36993,1675965623805, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36993, sessionid=0x1003a3dcfe40001 2023-02-09 18:00:25,253 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34863,1675965623892, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34863, sessionid=0x1003a3dcfe40003 2023-02-09 18:00:25,253 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 18:00:25,253 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 18:00:25,254 DEBUG [RS:2;jenkins-hbase4:34863] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,253 DEBUG [RS:0;jenkins-hbase4:36993] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,254 DEBUG [RS:1;jenkins-hbase4:34757] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 18:00:25,254 DEBUG [RS:0;jenkins-hbase4:36993] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36993,1675965623805' 2023-02-09 18:00:25,254 DEBUG [RS:0;jenkins-hbase4:36993] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 18:00:25,254 DEBUG [RS:2;jenkins-hbase4:34863] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34863,1675965623892' 2023-02-09 18:00:25,255 DEBUG [RS:2;jenkins-hbase4:34863] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 18:00:25,255 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 18:00:25,255 DEBUG [RS:0;jenkins-hbase4:36993] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 18:00:25,255 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 18:00:25,255 DEBUG [RS:1;jenkins-hbase4:34757] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:25,255 DEBUG [RS:2;jenkins-hbase4:34863] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 18:00:25,255 DEBUG [RS:1;jenkins-hbase4:34757] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34757,1675965623856' 2023-02-09 18:00:25,256 DEBUG [RS:1;jenkins-hbase4:34757] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 18:00:25,256 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 18:00:25,256 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 18:00:25,256 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 18:00:25,256 DEBUG [RS:2;jenkins-hbase4:34863] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:25,257 DEBUG [RS:1;jenkins-hbase4:34757] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 18:00:25,257 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 18:00:25,257 DEBUG [RS:0;jenkins-hbase4:36993] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,257 DEBUG [RS:0;jenkins-hbase4:36993] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36993,1675965623805' 2023-02-09 18:00:25,257 DEBUG [RS:0;jenkins-hbase4:36993] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 18:00:25,257 DEBUG [RS:2;jenkins-hbase4:34863] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34863,1675965623892' 2023-02-09 18:00:25,257 DEBUG [RS:2;jenkins-hbase4:34863] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 18:00:25,257 DEBUG [RS:1;jenkins-hbase4:34757] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 18:00:25,257 INFO [RS:1;jenkins-hbase4:34757] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 18:00:25,257 INFO [RS:1;jenkins-hbase4:34757] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 18:00:25,260 DEBUG [RS:2;jenkins-hbase4:34863] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 18:00:25,260 DEBUG [RS:0;jenkins-hbase4:36993] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 18:00:25,260 DEBUG [RS:2;jenkins-hbase4:34863] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 18:00:25,260 DEBUG [RS:0;jenkins-hbase4:36993] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 18:00:25,260 INFO [RS:2;jenkins-hbase4:34863] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 18:00:25,261 INFO [RS:0;jenkins-hbase4:36993] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 18:00:25,261 INFO [RS:2;jenkins-hbase4:34863] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 18:00:25,261 INFO [RS:0;jenkins-hbase4:36993] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 18:00:25,322 DEBUG [jenkins-hbase4:46015] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-09 18:00:25,325 DEBUG [jenkins-hbase4:46015] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-09 18:00:25,331 DEBUG [jenkins-hbase4:46015] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 18:00:25,331 DEBUG [jenkins-hbase4:46015] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 18:00:25,331 DEBUG [jenkins-hbase4:46015] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 18:00:25,331 DEBUG [jenkins-hbase4:46015] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 18:00:25,334 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36993,1675965623805, state=OPENING 2023-02-09 18:00:25,341 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-09 18:00:25,342 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:25,343 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-09 18:00:25,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36993,1675965623805}] 2023-02-09 18:00:25,369 INFO [RS:1;jenkins-hbase4:34757] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34757%2C1675965623856, suffix=, logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34757,1675965623856, archiveDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs, maxLogs=32 2023-02-09 18:00:25,369 INFO [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36993%2C1675965623805, suffix=, logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,36993,1675965623805, archiveDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs, maxLogs=32 2023-02-09 18:00:25,369 INFO [RS:2;jenkins-hbase4:34863] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34863%2C1675965623892, suffix=, logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34863,1675965623892, archiveDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs, maxLogs=32 2023-02-09 18:00:25,397 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK] 2023-02-09 18:00:25,398 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK] 2023-02-09 18:00:25,400 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK] 2023-02-09 18:00:25,400 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK] 2023-02-09 18:00:25,400 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK] 2023-02-09 18:00:25,401 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK] 2023-02-09 18:00:25,411 INFO [RS:1;jenkins-hbase4:34757] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34757,1675965623856/jenkins-hbase4.apache.org%2C34757%2C1675965623856.1675965625374 2023-02-09 18:00:25,411 DEBUG [RS:1;jenkins-hbase4:34757] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK], DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK], DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK]] 2023-02-09 18:00:25,415 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK] 2023-02-09 18:00:25,415 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK] 2023-02-09 18:00:25,416 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK] 2023-02-09 18:00:25,422 INFO [RS:2;jenkins-hbase4:34863] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34863,1675965623892/jenkins-hbase4.apache.org%2C34863%2C1675965623892.1675965625374 2023-02-09 18:00:25,424 DEBUG [RS:2;jenkins-hbase4:34863] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK], DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK], DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK]] 2023-02-09 18:00:25,426 INFO [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,36993,1675965623805/jenkins-hbase4.apache.org%2C36993%2C1675965623805.1675965625374 2023-02-09 18:00:25,426 DEBUG [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK], DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK], DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK]] 2023-02-09 18:00:25,533 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:25,535 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 18:00:25,539 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57342, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 18:00:25,553 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-09 18:00:25,553 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:25,557 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36993%2C1675965623805.meta, suffix=.meta, logDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,36993,1675965623805, archiveDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs, maxLogs=32 2023-02-09 18:00:25,579 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK] 2023-02-09 18:00:25,582 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK] 2023-02-09 18:00:25,583 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK] 2023-02-09 18:00:25,590 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,36993,1675965623805/jenkins-hbase4.apache.org%2C36993%2C1675965623805.meta.1675965625560.meta 2023-02-09 18:00:25,591 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43749,DS-a0396d2b-0b28-451c-9014-3c7f3064c2d6,DISK], DatanodeInfoWithStorage[127.0.0.1:44441,DS-a0cf1878-0ebd-4479-9583-f66a68830d81,DISK], DatanodeInfoWithStorage[127.0.0.1:45057,DS-549ef1fd-96f5-4bfe-8cb1-7effb23144a1,DISK]] 2023-02-09 18:00:25,591 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-09 18:00:25,592 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-09 18:00:25,610 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-09 18:00:25,616 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-09 18:00:25,621 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-09 18:00:25,621 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:25,621 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-09 18:00:25,621 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-09 18:00:25,624 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-09 18:00:25,626 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/info 2023-02-09 18:00:25,626 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/info 2023-02-09 18:00:25,626 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-09 18:00:25,627 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:25,628 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-09 18:00:25,629 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/rep_barrier 2023-02-09 18:00:25,629 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/rep_barrier 2023-02-09 18:00:25,630 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-09 18:00:25,630 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:25,631 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-09 18:00:25,632 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/table 2023-02-09 18:00:25,632 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/table 2023-02-09 18:00:25,633 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-09 18:00:25,633 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:25,635 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740 2023-02-09 18:00:25,638 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740 2023-02-09 18:00:25,641 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-09 18:00:25,644 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-09 18:00:25,646 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70441807, jitterRate=0.04966472089290619}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-09 18:00:25,646 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1675965625622Writing region info on filesystem at 1675965625622Initializing all the Stores at 1675965625624 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965625624Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675965625624Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965625624Cleaning up temporary data from old regions at 1675965625640 (+16 ms)Cleaning up detritus from prior splits at 1675965625640Running coprocessor post-open hooks at 1675965625646 (+6 ms)Region opened successfully at 1675965625646 2023-02-09 18:00:25,661 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1675965625525 2023-02-09 18:00:25,679 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-09 18:00:25,680 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-09 18:00:25,680 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36993,1675965623805, state=OPEN 2023-02-09 18:00:25,683 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-09 18:00:25,683 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-09 18:00:25,687 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-09 18:00:25,688 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36993,1675965623805 in 336 msec 2023-02-09 18:00:25,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-09 18:00:25,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 536 msec 2023-02-09 18:00:25,698 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 798 msec 2023-02-09 18:00:25,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1675965623938, completionTime=-1 2023-02-09 18:00:25,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-09 18:00:25,698 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 18:00:25,765 DEBUG [hconnection-0x3a10f490-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 18:00:25,768 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57348, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 18:00:25,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 18:00:25,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675965685786 2023-02-09 18:00:25,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675965745786 2023-02-09 18:00:25,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 87 msec 2023-02-09 18:00:25,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1675965622645-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1675965622645-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1675965622645-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46015, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:25,824 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-09 18:00:25,833 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-09 18:00:25,834 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-09 18:00:25,844 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-09 18:00:25,847 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 18:00:25,849 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 18:00:25,873 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/.tmp/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:25,876 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/.tmp/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a empty. 2023-02-09 18:00:25,877 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/.tmp/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:25,877 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-09 18:00:25,914 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-09 18:00:25,917 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => ba90e0fd385d9c5714e7df5263d3442a, NAME => 'hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/.tmp 2023-02-09 18:00:25,936 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:25,936 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing ba90e0fd385d9c5714e7df5263d3442a, disabling compactions & flushes 2023-02-09 18:00:25,936 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:25,936 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:25,937 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. after waiting 0 ms 2023-02-09 18:00:25,937 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:25,937 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:25,937 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for ba90e0fd385d9c5714e7df5263d3442a: Waiting for close lock at 1675965625936Disabling compacts and flushes for region at 1675965625936Disabling writes for close at 1675965625937 (+1 ms)Writing region close event to WAL at 1675965625937Closed at 1675965625937 2023-02-09 18:00:25,941 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 18:00:25,957 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675965625944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675965625944"}]},"ts":"1675965625944"} 2023-02-09 18:00:25,985 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 18:00:25,987 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 18:00:25,992 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675965625987"}]},"ts":"1675965625987"} 2023-02-09 18:00:25,997 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-09 18:00:26,002 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-09 18:00:26,003 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 18:00:26,003 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 18:00:26,003 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 18:00:26,003 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 18:00:26,006 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ba90e0fd385d9c5714e7df5263d3442a, ASSIGN}] 2023-02-09 18:00:26,009 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ba90e0fd385d9c5714e7df5263d3442a, ASSIGN 2023-02-09 18:00:26,011 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ba90e0fd385d9c5714e7df5263d3442a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36993,1675965623805; forceNewPlan=false, retain=false 2023-02-09 18:00:26,164 INFO [jenkins-hbase4:46015] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 18:00:26,166 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ba90e0fd385d9c5714e7df5263d3442a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:26,166 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675965626165"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675965626165"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675965626165"}]},"ts":"1675965626165"} 2023-02-09 18:00:26,170 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure ba90e0fd385d9c5714e7df5263d3442a, server=jenkins-hbase4.apache.org,36993,1675965623805}] 2023-02-09 18:00:26,333 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,334 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ba90e0fd385d9c5714e7df5263d3442a, NAME => 'hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.', STARTKEY => '', ENDKEY => ''} 2023-02-09 18:00:26,335 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,335 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:26,335 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,335 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,337 INFO [StoreOpener-ba90e0fd385d9c5714e7df5263d3442a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,340 DEBUG [StoreOpener-ba90e0fd385d9c5714e7df5263d3442a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/info 2023-02-09 18:00:26,340 DEBUG [StoreOpener-ba90e0fd385d9c5714e7df5263d3442a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/info 2023-02-09 18:00:26,340 INFO [StoreOpener-ba90e0fd385d9c5714e7df5263d3442a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba90e0fd385d9c5714e7df5263d3442a columnFamilyName info 2023-02-09 18:00:26,341 INFO [StoreOpener-ba90e0fd385d9c5714e7df5263d3442a-1] regionserver.HStore(310): Store=ba90e0fd385d9c5714e7df5263d3442a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:26,343 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,344 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,349 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,353 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 18:00:26,354 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ba90e0fd385d9c5714e7df5263d3442a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69438430, jitterRate=0.03471323847770691}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 18:00:26,354 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ba90e0fd385d9c5714e7df5263d3442a: Running coprocessor pre-open hook at 1675965626335Writing region info on filesystem at 1675965626335Initializing all the Stores at 1675965626337 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965626337Cleaning up temporary data from old regions at 1675965626345 (+8 ms)Cleaning up detritus from prior splits at 1675965626346 (+1 ms)Running coprocessor post-open hooks at 1675965626354 (+8 ms)Region opened successfully at 1675965626354 2023-02-09 18:00:26,356 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a., pid=6, masterSystemTime=1675965626324 2023-02-09 18:00:26,360 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,360 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,362 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ba90e0fd385d9c5714e7df5263d3442a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:26,362 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675965626361"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675965626361"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675965626361"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675965626361"}]},"ts":"1675965626361"} 2023-02-09 18:00:26,371 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-09 18:00:26,371 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure ba90e0fd385d9c5714e7df5263d3442a, server=jenkins-hbase4.apache.org,36993,1675965623805 in 197 msec 2023-02-09 18:00:26,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-09 18:00:26,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ba90e0fd385d9c5714e7df5263d3442a, ASSIGN in 365 msec 2023-02-09 18:00:26,377 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 18:00:26,378 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675965626377"}]},"ts":"1675965626377"} 2023-02-09 18:00:26,380 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-09 18:00:26,385 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 18:00:26,387 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 550 msec 2023-02-09 18:00:26,447 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-09 18:00:26,449 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-09 18:00:26,449 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:26,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-09 18:00:26,498 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-09 18:00:26,504 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-02-09 18:00:26,515 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-09 18:00:26,528 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-09 18:00:26,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-09 18:00:26,541 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-09 18:00:26,545 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-09 18:00:26,545 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.594sec 2023-02-09 18:00:26,548 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-09 18:00:26,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-09 18:00:26,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-09 18:00:26,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1675965622645-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-09 18:00:26,553 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1675965622645-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-09 18:00:26,562 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-09 18:00:26,621 DEBUG [Listener at localhost/38227] zookeeper.ReadOnlyZKClient(139): Connect 0x3e2642e8 to 127.0.0.1:49196 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:26,626 DEBUG [Listener at localhost/38227] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@261e807e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:26,641 DEBUG [hconnection-0x450b1e73-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 18:00:26,653 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57360, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 18:00:26,662 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:26,663 DEBUG [Listener at localhost/38227] zookeeper.ReadOnlyZKClient(139): Connect 0x7152532e to 127.0.0.1:49196 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:26,704 DEBUG [ReadOnlyZKClient-127.0.0.1:49196@0x7152532e] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@add00bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:26,740 DEBUG [Listener at localhost/38227] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 18:00:26,744 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57374, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 18:00:26,745 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36993] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36993,1675965623805' ***** 2023-02-09 18:00:26,745 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36993] regionserver.HRegionServer(2310): STOPPED: Called by admin client org.apache.hadoop.hbase.client.AsyncConnectionImpl@1a9e5881 2023-02-09 18:00:26,746 INFO [RS:0;jenkins-hbase4:36993] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 18:00:26,746 INFO [RS:0;jenkins-hbase4:36993] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 18:00:26,746 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 18:00:26,747 INFO [RS:0;jenkins-hbase4:36993] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 18:00:26,747 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(3304): Received CLOSE for ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,748 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:26,749 DEBUG [RS:0;jenkins-hbase4:36993] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79795eee to 127.0.0.1:49196 2023-02-09 18:00:26,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ba90e0fd385d9c5714e7df5263d3442a, disabling compactions & flushes 2023-02-09 18:00:26,749 DEBUG [RS:0;jenkins-hbase4:36993] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,750 INFO [RS:0;jenkins-hbase4:36993] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 18:00:26,750 INFO [RS:0;jenkins-hbase4:36993] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 18:00:26,750 INFO [RS:0;jenkins-hbase4:36993] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 18:00:26,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. after waiting 0 ms 2023-02-09 18:00:26,750 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-09 18:00:26,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,750 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-09 18:00:26,751 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, ba90e0fd385d9c5714e7df5263d3442a=hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a.} 2023-02-09 18:00:26,751 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-09 18:00:26,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ba90e0fd385d9c5714e7df5263d3442a 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-09 18:00:26,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-09 18:00:26,752 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-09 18:00:26,752 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-09 18:00:26,752 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-09 18:00:26,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-02-09 18:00:26,753 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1505): Waiting on 1588230740, ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,753 DEBUG [Listener at localhost/38227] client.ConnectionUtils(586): Start fetching master stub from registry 2023-02-09 18:00:26,754 DEBUG [ReadOnlyZKClient-127.0.0.1:49196@0x7152532e] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:26,759 DEBUG [ReadOnlyZKClient-127.0.0.1:49196@0x7152532e] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@222976ff 2023-02-09 18:00:26,766 DEBUG [ReadOnlyZKClient-127.0.0.1:49196@0x7152532e] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-09 18:00:26,770 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50644, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-09 18:00:26,770 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46015] master.MasterRpcServices(1601): Client=jenkins//172.31.14.131 stop 2023-02-09 18:00:26,770 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46015] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46015,1675965622645' ***** 2023-02-09 18:00:26,770 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46015] regionserver.HRegionServer(2310): STOPPED: Stopped by RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46015 2023-02-09 18:00:26,771 DEBUG [M:0;jenkins-hbase4:46015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18a5b18f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:26,771 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46015,1675965622645 2023-02-09 18:00:26,774 DEBUG [M:0;jenkins-hbase4:46015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d0608b6 to 127.0.0.1:49196 2023-02-09 18:00:26,774 DEBUG [M:0;jenkins-hbase4:46015] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,774 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46015,1675965622645; all regions closed. 2023-02-09 18:00:26,774 DEBUG [M:0;jenkins-hbase4:46015] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,775 DEBUG [M:0;jenkins-hbase4:46015] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-09 18:00:26,775 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:26,775 DEBUG [M:0;jenkins-hbase4:46015] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-09 18:00:26,775 INFO [M:0;jenkins-hbase4:46015] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-09 18:00:26,775 INFO [M:0;jenkins-hbase4:46015] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-09 18:00:26,775 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:26,776 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-09 18:00:26,775 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1675965625021] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1675965625021,5,FailOnTimeoutGroup] 2023-02-09 18:00:26,775 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1675965625022] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1675965625022,5,FailOnTimeoutGroup] 2023-02-09 18:00:26,776 INFO [M:0;jenkins-hbase4:46015] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-09 18:00:26,776 DEBUG [M:0;jenkins-hbase4:46015] master.HMaster(1502): Stopping service threads 2023-02-09 18:00:26,776 INFO [M:0;jenkins-hbase4:46015] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-09 18:00:26,777 INFO [M:0;jenkins-hbase4:46015] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 18:00:26,776 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:26,775 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:26,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:26,780 ERROR [M:0;jenkins-hbase4:46015] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] 2023-02-09 18:00:26,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:26,777 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:26,781 INFO [M:0;jenkins-hbase4:46015] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 18:00:26,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:26,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:26,781 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-09 18:00:26,781 INFO [Listener at localhost/38227] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/38227. 2023-02-09 18:00:26,781 DEBUG [M:0;jenkins-hbase4:46015] zookeeper.ZKUtil(398): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-09 18:00:26,781 DEBUG [Listener at localhost/38227] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2.tearDown(TestAsyncClusterAdminApi2.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-02-09 18:00:26,781 WARN [M:0;jenkins-hbase4:46015] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-09 18:00:26,781 INFO [M:0;jenkins-hbase4:46015] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 18:00:26,782 INFO [M:0;jenkins-hbase4:46015] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-09 18:00:26,784 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-09 18:00:26,784 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:26,784 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:26,784 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-09 18:00:26,784 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:26,785 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.07 KB heapSize=29.55 KB 2023-02-09 18:00:26,790 DEBUG [Listener at localhost/38227] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,791 DEBUG [Listener at localhost/38227] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7152532e to 127.0.0.1:49196 2023-02-09 18:00:26,792 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-09 18:00:26,795 DEBUG [Listener at localhost/38227] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e2642e8 to 127.0.0.1:49196 2023-02-09 18:00:26,795 DEBUG [Listener at localhost/38227] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,796 DEBUG [Listener at localhost/38227] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-09 18:00:26,797 INFO [Listener at localhost/38227] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34757,1675965623856' ***** 2023-02-09 18:00:26,797 INFO [Listener at localhost/38227] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-09 18:00:26,797 INFO [Listener at localhost/38227] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34863,1675965623892' ***** 2023-02-09 18:00:26,797 INFO [Listener at localhost/38227] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-09 18:00:26,797 INFO [RS:1;jenkins-hbase4:34757] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 18:00:26,797 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 18:00:26,798 INFO [RS:2;jenkins-hbase4:34863] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 18:00:26,798 INFO [RS:1;jenkins-hbase4:34757] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 18:00:26,798 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 18:00:26,798 INFO [RS:1;jenkins-hbase4:34757] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 18:00:26,798 INFO [RS:2;jenkins-hbase4:34863] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 18:00:26,798 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:26,798 INFO [RS:2;jenkins-hbase4:34863] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 18:00:26,798 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:26,799 DEBUG [RS:1;jenkins-hbase4:34757] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c3b4380 to 127.0.0.1:49196 2023-02-09 18:00:26,799 DEBUG [RS:2;jenkins-hbase4:34863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54d0eb27 to 127.0.0.1:49196 2023-02-09 18:00:26,799 DEBUG [RS:1;jenkins-hbase4:34757] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,799 DEBUG [RS:2;jenkins-hbase4:34863] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,799 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34863,1675965623892; all regions closed. 2023-02-09 18:00:26,799 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34757,1675965623856; all regions closed. 2023-02-09 18:00:26,812 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/WALs/jenkins-hbase4.apache.org,34863,1675965623892/jenkins-hbase4.apache.org%2C34863%2C1675965623892.1675965625374 not finished, retry = 0 2023-02-09 18:00:26,814 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:26,816 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:26,818 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:26,822 DEBUG [RS:1;jenkins-hbase4:34757] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs 2023-02-09 18:00:26,822 INFO [RS:1;jenkins-hbase4:34757] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34757%2C1675965623856:(num 1675965625374) 2023-02-09 18:00:26,822 DEBUG [RS:1;jenkins-hbase4:34757] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,822 INFO [RS:1;jenkins-hbase4:34757] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:26,822 INFO [RS:1;jenkins-hbase4:34757] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-09 18:00:26,823 INFO [RS:1;jenkins-hbase4:34757] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 18:00:26,823 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:26,823 INFO [RS:1;jenkins-hbase4:34757] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 18:00:26,823 INFO [RS:1;jenkins-hbase4:34757] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 18:00:26,824 INFO [RS:1;jenkins-hbase4:34757] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34757 2023-02-09 18:00:26,864 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/.tmp/info/49357d058bb345409172c5716c60c6a4 2023-02-09 18:00:26,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/.tmp/info/68d08cfb5be34b5987b5c37149e7ff49 2023-02-09 18:00:26,869 INFO [M:0;jenkins-hbase4:46015] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/97e7f786d1664d06b56f8bd2d7b1e7f8 2023-02-09 18:00:26,905 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:26,905 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:26,905 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34757,1675965623856 2023-02-09 18:00:26,906 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:26,906 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3666bbbe rejected from java.util.concurrent.ThreadPoolExecutor@325939aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:26,905 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:26,905 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:26,906 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:26,906 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@45de0a36 rejected from java.util.concurrent.ThreadPoolExecutor@325939aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:26,911 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/97e7f786d1664d06b56f8bd2d7b1e7f8 as hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/97e7f786d1664d06b56f8bd2d7b1e7f8 2023-02-09 18:00:26,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/.tmp/info/68d08cfb5be34b5987b5c37149e7ff49 as hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/info/68d08cfb5be34b5987b5c37149e7ff49 2023-02-09 18:00:26,919 DEBUG [RS:2;jenkins-hbase4:34863] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs 2023-02-09 18:00:26,919 INFO [RS:2;jenkins-hbase4:34863] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34863%2C1675965623892:(num 1675965625374) 2023-02-09 18:00:26,919 DEBUG [RS:2;jenkins-hbase4:34863] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:26,919 INFO [RS:2;jenkins-hbase4:34863] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:26,920 INFO [RS:2;jenkins-hbase4:34863] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-09 18:00:26,920 INFO [RS:2;jenkins-hbase4:34863] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 18:00:26,920 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:26,920 INFO [RS:2;jenkins-hbase4:34863] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 18:00:26,920 INFO [RS:2;jenkins-hbase4:34863] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 18:00:26,921 INFO [RS:2;jenkins-hbase4:34863] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34863 2023-02-09 18:00:26,928 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:26,928 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34863,1675965623892 2023-02-09 18:00:26,928 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@62467a9e rejected from java.util.concurrent.ThreadPoolExecutor@6679f20e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:26,930 INFO [M:0;jenkins-hbase4:46015] regionserver.HStore(1080): Added hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/97e7f786d1664d06b56f8bd2d7b1e7f8, entries=8, sequenceid=66, filesize=6.3 K 2023-02-09 18:00:26,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/info/68d08cfb5be34b5987b5c37149e7ff49, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 18:00:26,935 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/.tmp/table/841febe0ffa54ce2b4de504c0d51a057 2023-02-09 18:00:26,935 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegion(2947): Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 151ms, sequenceid=66, compaction requested=false 2023-02-09 18:00:26,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for ba90e0fd385d9c5714e7df5263d3442a in 184ms, sequenceid=6, compaction requested=false 2023-02-09 18:00:26,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-09 18:00:26,939 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:26,939 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1675965626782Disabling compacts and flushes for region at 1675965626782Disabling writes for close at 1675965626784 (+2 ms)Obtaining lock to block concurrent updates at 1675965626785 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1675965626785Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=24646, getHeapSize=30248, getOffHeapSize=0, getCellsCount=71 at 1675965626785Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1675965626790 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1675965626791 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1675965626818 (+27 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1675965626822 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1675965626913 (+91 ms)Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 151ms, sequenceid=66, compaction requested=false at 1675965626936 (+23 ms)Writing region close event to WAL at 1675965626939 (+3 ms)Closed at 1675965626939 2023-02-09 18:00:26,946 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:26,946 INFO [M:0;jenkins-hbase4:46015] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-09 18:00:26,947 INFO [M:0;jenkins-hbase4:46015] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46015 2023-02-09 18:00:26,953 DEBUG [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1505): Waiting on 1588230740, ba90e0fd385d9c5714e7df5263d3442a 2023-02-09 18:00:26,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/namespace/ba90e0fd385d9c5714e7df5263d3442a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-09 18:00:26,953 DEBUG [M:0;jenkins-hbase4:46015] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46015,1675965622645 already deleted, retry=false 2023-02-09 18:00:26,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ba90e0fd385d9c5714e7df5263d3442a: Waiting for close lock at 1675965626749Running coprocessor pre-close hooks at 1675965626749Disabling compacts and flushes for region at 1675965626749Disabling writes for close at 1675965626750 (+1 ms)Obtaining lock to block concurrent updates at 1675965626752 (+2 ms)Preparing flush snapshotting stores in ba90e0fd385d9c5714e7df5263d3442a at 1675965626752Finished memstore snapshotting hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1675965626761 (+9 ms)Flushing stores of hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. at 1675965626763 (+2 ms)Flushing ba90e0fd385d9c5714e7df5263d3442a/info: creating writer at 1675965626767 (+4 ms)Flushing ba90e0fd385d9c5714e7df5263d3442a/info: appending metadata at 1675965626817 (+50 ms)Flushing ba90e0fd385d9c5714e7df5263d3442a/info: closing flushed file at 1675965626822 (+5 ms)Flushing ba90e0fd385d9c5714e7df5263d3442a/info: reopening flushed file at 1675965626915 (+93 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for ba90e0fd385d9c5714e7df5263d3442a in 184ms, sequenceid=6, compaction requested=false at 1675965626936 (+21 ms)Writing region close event to WAL at 1675965626943 (+7 ms)Running coprocessor post-close hooks at 1675965626955 (+12 ms)Closed at 1675965626956 (+1 ms) 2023-02-09 18:00:26,956 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/.tmp/info/49357d058bb345409172c5716c60c6a4 as hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/info/49357d058bb345409172c5716c60c6a4 2023-02-09 18:00:26,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1675965625833.ba90e0fd385d9c5714e7df5263d3442a. 2023-02-09 18:00:26,965 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/info/49357d058bb345409172c5716c60c6a4, entries=10, sequenceid=9, filesize=5.9 K 2023-02-09 18:00:26,967 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/.tmp/table/841febe0ffa54ce2b4de504c0d51a057 as hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/table/841febe0ffa54ce2b4de504c0d51a057 2023-02-09 18:00:26,977 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/table/841febe0ffa54ce2b4de504c0d51a057, entries=2, sequenceid=9, filesize=4.7 K 2023-02-09 18:00:26,979 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 227ms, sequenceid=9, compaction requested=false 2023-02-09 18:00:26,979 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-09 18:00:26,988 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-02-09 18:00:26,989 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-09 18:00:26,989 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-09 18:00:26,990 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1675965626751Running coprocessor pre-close hooks at 1675965626751Disabling compacts and flushes for region at 1675965626751Disabling writes for close at 1675965626752 (+1 ms)Obtaining lock to block concurrent updates at 1675965626752Preparing flush snapshotting stores in 1588230740 at 1675965626752Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1290, getHeapSize=2912, getOffHeapSize=0, getCellsCount=12 at 1675965626761 (+9 ms)Flushing stores of hbase:meta,,1.1588230740 at 1675965626763 (+2 ms)Flushing 1588230740/info: creating writer at 1675965626767 (+4 ms)Flushing 1588230740/info: appending metadata at 1675965626817 (+50 ms)Flushing 1588230740/info: closing flushed file at 1675965626822 (+5 ms)Flushing 1588230740/table: creating writer at 1675965626909 (+87 ms)Flushing 1588230740/table: appending metadata at 1675965626916 (+7 ms)Flushing 1588230740/table: closing flushed file at 1675965626916Flushing 1588230740/info: reopening flushed file at 1675965626957 (+41 ms)Flushing 1588230740/table: reopening flushed file at 1675965626968 (+11 ms)Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 227ms, sequenceid=9, compaction requested=false at 1675965626979 (+11 ms)Writing region close event to WAL at 1675965626983 (+4 ms)Running coprocessor post-close hooks at 1675965626989 (+6 ms)Closed at 1675965626989 2023-02-09 18:00:26,990 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-09 18:00:27,154 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36993,1675965623805; all regions closed. 2023-02-09 18:00:27,162 DEBUG [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs 2023-02-09 18:00:27,162 INFO [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36993%2C1675965623805.meta:.meta(num 1675965625560) 2023-02-09 18:00:27,169 DEBUG [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/oldWALs 2023-02-09 18:00:27,169 INFO [RS:0;jenkins-hbase4:36993] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36993%2C1675965623805:(num 1675965625374) 2023-02-09 18:00:27,169 DEBUG [RS:0;jenkins-hbase4:36993] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:27,169 INFO [RS:0;jenkins-hbase4:36993] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:27,170 INFO [RS:0;jenkins-hbase4:36993] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-09 18:00:27,170 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:27,170 INFO [RS:0;jenkins-hbase4:36993] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36993 2023-02-09 18:00:27,174 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36993,1675965623805 2023-02-09 18:00:27,174 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@10a443bc rejected from java.util.concurrent.ThreadPoolExecutor@149bc570[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,356 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,356 INFO [RS:0;jenkins-hbase4:36993] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36993,1675965623805; zookeeper connection closed. 2023-02-09 18:00:27,356 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5209fe1a rejected from java.util.concurrent.ThreadPoolExecutor@149bc570[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,357 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:36993-0x1003a3dcfe40001, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,357 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6e8c0f38] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6e8c0f38 2023-02-09 18:00:27,357 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2af81761 rejected from java.util.concurrent.ThreadPoolExecutor@149bc570[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,456 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,456 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46015,1675965622645; zookeeper connection closed. 2023-02-09 18:00:27,456 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@da5ff77 rejected from java.util.concurrent.ThreadPoolExecutor@7bb6b5f6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,457 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1003a3dcfe40000, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,457 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c994d9f rejected from java.util.concurrent.ThreadPoolExecutor@7bb6b5f6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,556 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,556 INFO [RS:2;jenkins-hbase4:34863] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34863,1675965623892; zookeeper connection closed. 2023-02-09 18:00:27,557 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43d157dd rejected from java.util.concurrent.ThreadPoolExecutor@6679f20e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,557 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34863-0x1003a3dcfe40003, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,557 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@39a27c3a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@39a27c3a 2023-02-09 18:00:27,557 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b522e7d rejected from java.util.concurrent.ThreadPoolExecutor@6679f20e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,657 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,657 INFO [RS:1;jenkins-hbase4:34757] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34757,1675965623856; zookeeper connection closed. 2023-02-09 18:00:27,657 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3d090777 rejected from java.util.concurrent.ThreadPoolExecutor@325939aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,657 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@474184ba] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@474184ba 2023-02-09 18:00:27,657 DEBUG [Listener at localhost/38227-EventThread] zookeeper.ZKWatcher(600): regionserver:34757-0x1003a3dcfe40002, quorum=127.0.0.1:49196, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:27,657 ERROR [Listener at localhost/38227-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@22ce69a8 rejected from java.util.concurrent.ThreadPoolExecutor@325939aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:27,657 INFO [Listener at localhost/38227] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-09 18:00:27,658 WARN [Listener at localhost/38227] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 18:00:27,661 INFO [Listener at localhost/38227] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:27,764 WARN [BP-915768045-172.31.14.131-1675965619531 heartbeating to localhost/127.0.0.1:44505] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 18:00:27,764 WARN [BP-915768045-172.31.14.131-1675965619531 heartbeating to localhost/127.0.0.1:44505] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-915768045-172.31.14.131-1675965619531 (Datanode Uuid c9d26d5c-ecc3-44b5-b864-eece467a5b9b) service to localhost/127.0.0.1:44505 2023-02-09 18:00:27,766 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/dfs/data/data5/current/BP-915768045-172.31.14.131-1675965619531] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:27,766 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/dfs/data/data6/current/BP-915768045-172.31.14.131-1675965619531] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:27,766 WARN [Listener at localhost/38227] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 18:00:27,769 INFO [Listener at localhost/38227] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:27,871 WARN [BP-915768045-172.31.14.131-1675965619531 heartbeating to localhost/127.0.0.1:44505] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 18:00:27,871 WARN [BP-915768045-172.31.14.131-1675965619531 heartbeating to localhost/127.0.0.1:44505] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-915768045-172.31.14.131-1675965619531 (Datanode Uuid 6dd596e7-dc0f-4c35-80ba-5c41aa63abea) service to localhost/127.0.0.1:44505 2023-02-09 18:00:27,872 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/dfs/data/data3/current/BP-915768045-172.31.14.131-1675965619531] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:27,872 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/dfs/data/data4/current/BP-915768045-172.31.14.131-1675965619531] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:27,873 WARN [Listener at localhost/38227] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 18:00:27,876 INFO [Listener at localhost/38227] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:27,978 WARN [BP-915768045-172.31.14.131-1675965619531 heartbeating to localhost/127.0.0.1:44505] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 18:00:27,978 WARN [BP-915768045-172.31.14.131-1675965619531 heartbeating to localhost/127.0.0.1:44505] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-915768045-172.31.14.131-1675965619531 (Datanode Uuid 82264b30-6c2f-4134-9b35-60a3df5f7b06) service to localhost/127.0.0.1:44505 2023-02-09 18:00:27,978 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/dfs/data/data1/current/BP-915768045-172.31.14.131-1675965619531] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:27,979 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/cluster_74d8ae81-5710-f880-a43d-8a9988bd5886/dfs/data/data2/current/BP-915768045-172.31.14.131-1675965619531] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:28,003 INFO [Listener at localhost/38227] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:28,117 INFO [Listener at localhost/38227] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-09 18:00:28,154 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-09 18:00:28,167 INFO [Listener at localhost/38227] hbase.ResourceChecker(175): after: client.TestAsyncClusterAdminApi2#testStop Thread=79 (was 8) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:44505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:44505 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:44505 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:44505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:44505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/38227 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@d1663db java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:44505 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'NameNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:44505 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:44505 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:44505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=488 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=3263 (was 3877) 2023-02-09 18:00:28,178 INFO [Listener at localhost/38227] hbase.ResourceChecker(147): before: client.TestAsyncClusterAdminApi2#testShutdown Thread=79, OpenFileDescriptor=488, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=167, AvailableMemoryMB=3263 2023-02-09 18:00:28,178 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-09 18:00:28,179 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/hadoop.log.dir so I do NOT create it in target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd 2023-02-09 18:00:28,179 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/71c5fef1-174e-5044-af3c-3b3741f0847c/hadoop.tmp.dir so I do NOT create it in target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd 2023-02-09 18:00:28,179 INFO [Listener at localhost/38227] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e, deleteOnExit=true 2023-02-09 18:00:28,179 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-09 18:00:28,179 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/test.cache.data in system properties and HBase conf 2023-02-09 18:00:28,179 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/hadoop.tmp.dir in system properties and HBase conf 2023-02-09 18:00:28,180 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/hadoop.log.dir in system properties and HBase conf 2023-02-09 18:00:28,180 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-09 18:00:28,180 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-09 18:00:28,180 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-09 18:00:28,181 DEBUG [Listener at localhost/38227] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-09 18:00:28,181 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-09 18:00:28,181 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-09 18:00:28,182 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-09 18:00:28,182 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-09 18:00:28,182 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-09 18:00:28,182 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-09 18:00:28,183 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-09 18:00:28,183 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-09 18:00:28,183 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-09 18:00:28,183 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/nfs.dump.dir in system properties and HBase conf 2023-02-09 18:00:28,183 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/java.io.tmpdir in system properties and HBase conf 2023-02-09 18:00:28,184 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-09 18:00:28,184 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-09 18:00:28,184 INFO [Listener at localhost/38227] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-09 18:00:28,189 WARN [Listener at localhost/38227] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-09 18:00:28,190 WARN [Listener at localhost/38227] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-09 18:00:28,236 WARN [Listener at localhost/38227] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:28,239 INFO [Listener at localhost/38227] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:28,246 INFO [Listener at localhost/38227] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/java.io.tmpdir/Jetty_localhost_35785_hdfs____vqwn7g/webapp 2023-02-09 18:00:28,346 INFO [Listener at localhost/38227] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35785 2023-02-09 18:00:28,351 WARN [Listener at localhost/38227] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-09 18:00:28,351 WARN [Listener at localhost/38227] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-09 18:00:28,392 WARN [Listener at localhost/35575] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:28,403 WARN [Listener at localhost/35575] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 18:00:28,406 WARN [Listener at localhost/35575] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:28,407 INFO [Listener at localhost/35575] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:28,412 INFO [Listener at localhost/35575] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/java.io.tmpdir/Jetty_localhost_45847_datanode____vxstq9/webapp 2023-02-09 18:00:28,503 INFO [Listener at localhost/35575] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45847 2023-02-09 18:00:28,510 WARN [Listener at localhost/35013] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:28,529 WARN [Listener at localhost/35013] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 18:00:28,532 WARN [Listener at localhost/35013] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:28,533 INFO [Listener at localhost/35013] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:28,536 INFO [Listener at localhost/35013] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/java.io.tmpdir/Jetty_localhost_41885_datanode____.t23d6p/webapp 2023-02-09 18:00:28,628 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd7d9eb3ee7440837: Processing first storage report for DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4 from datanode d5ee1a81-9006-40a4-a9c9-30ebebb899e8 2023-02-09 18:00:28,629 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd7d9eb3ee7440837: from storage DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4 node DatanodeRegistration(127.0.0.1:46517, datanodeUuid=d5ee1a81-9006-40a4-a9c9-30ebebb899e8, infoPort=36017, infoSecurePort=0, ipcPort=35013, storageInfo=lv=-57;cid=testClusterID;nsid=491063105;c=1675965628192), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-09 18:00:28,629 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd7d9eb3ee7440837: Processing first storage report for DS-1d7932c3-9385-46ec-bfc8-3aa5add577a5 from datanode d5ee1a81-9006-40a4-a9c9-30ebebb899e8 2023-02-09 18:00:28,629 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd7d9eb3ee7440837: from storage DS-1d7932c3-9385-46ec-bfc8-3aa5add577a5 node DatanodeRegistration(127.0.0.1:46517, datanodeUuid=d5ee1a81-9006-40a4-a9c9-30ebebb899e8, infoPort=36017, infoSecurePort=0, ipcPort=35013, storageInfo=lv=-57;cid=testClusterID;nsid=491063105;c=1675965628192), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:28,648 INFO [Listener at localhost/35013] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41885 2023-02-09 18:00:28,655 WARN [Listener at localhost/41307] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:28,674 WARN [Listener at localhost/41307] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 18:00:28,677 WARN [Listener at localhost/41307] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 18:00:28,678 INFO [Listener at localhost/41307] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 18:00:28,682 INFO [Listener at localhost/41307] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/java.io.tmpdir/Jetty_localhost_46385_datanode____h5nv6d/webapp 2023-02-09 18:00:28,757 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc414e5e0f3c8953a: Processing first storage report for DS-39d61341-2c1f-4579-8962-4d63660d2dae from datanode e8e292da-c79b-4bc6-943a-d8433902573f 2023-02-09 18:00:28,757 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc414e5e0f3c8953a: from storage DS-39d61341-2c1f-4579-8962-4d63660d2dae node DatanodeRegistration(127.0.0.1:34185, datanodeUuid=e8e292da-c79b-4bc6-943a-d8433902573f, infoPort=33973, infoSecurePort=0, ipcPort=41307, storageInfo=lv=-57;cid=testClusterID;nsid=491063105;c=1675965628192), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:28,758 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc414e5e0f3c8953a: Processing first storage report for DS-f7192413-d269-4e36-b6cc-943297d9fb25 from datanode e8e292da-c79b-4bc6-943a-d8433902573f 2023-02-09 18:00:28,758 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc414e5e0f3c8953a: from storage DS-f7192413-d269-4e36-b6cc-943297d9fb25 node DatanodeRegistration(127.0.0.1:34185, datanodeUuid=e8e292da-c79b-4bc6-943a-d8433902573f, infoPort=33973, infoSecurePort=0, ipcPort=41307, storageInfo=lv=-57;cid=testClusterID;nsid=491063105;c=1675965628192), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:28,784 INFO [Listener at localhost/41307] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46385 2023-02-09 18:00:28,792 WARN [Listener at localhost/37495] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 18:00:28,891 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5edb9cb26101df8b: Processing first storage report for DS-fe179a80-d98a-414a-ae37-3904651655d9 from datanode 1820aa52-cc4c-4872-8f61-44f60519f804 2023-02-09 18:00:28,891 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5edb9cb26101df8b: from storage DS-fe179a80-d98a-414a-ae37-3904651655d9 node DatanodeRegistration(127.0.0.1:46031, datanodeUuid=1820aa52-cc4c-4872-8f61-44f60519f804, infoPort=34033, infoSecurePort=0, ipcPort=37495, storageInfo=lv=-57;cid=testClusterID;nsid=491063105;c=1675965628192), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:28,891 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5edb9cb26101df8b: Processing first storage report for DS-aab34d76-e6ed-4145-9690-1566dd16340c from datanode 1820aa52-cc4c-4872-8f61-44f60519f804 2023-02-09 18:00:28,891 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5edb9cb26101df8b: from storage DS-aab34d76-e6ed-4145-9690-1566dd16340c node DatanodeRegistration(127.0.0.1:46031, datanodeUuid=1820aa52-cc4c-4872-8f61-44f60519f804, infoPort=34033, infoSecurePort=0, ipcPort=37495, storageInfo=lv=-57;cid=testClusterID;nsid=491063105;c=1675965628192), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 18:00:28,900 DEBUG [Listener at localhost/37495] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd 2023-02-09 18:00:28,904 INFO [Listener at localhost/37495] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/zookeeper_0, clientPort=50070, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-09 18:00:28,905 INFO [Listener at localhost/37495] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50070 2023-02-09 18:00:28,905 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:28,906 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:28,927 INFO [Listener at localhost/37495] util.FSUtils(479): Created version file at hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718 with version=8 2023-02-09 18:00:28,927 INFO [Listener at localhost/37495] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:44505/user/jenkins/test-data/32116f27-747b-56a7-a194-c196f81b3ba6/hbase-staging 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:28,929 INFO [Listener at localhost/37495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:28,932 INFO [Listener at localhost/37495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46519 2023-02-09 18:00:28,932 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:28,933 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:28,934 INFO [Listener at localhost/37495] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46519 connecting to ZooKeeper ensemble=127.0.0.1:50070 2023-02-09 18:00:28,941 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:465190x0, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:28,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46519-0x1003a3debac0000 connected 2023-02-09 18:00:28,963 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:28,963 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:28,964 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:28,965 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46519 2023-02-09 18:00:28,965 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46519 2023-02-09 18:00:28,965 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46519 2023-02-09 18:00:28,968 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46519 2023-02-09 18:00:28,968 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46519 2023-02-09 18:00:28,969 INFO [Listener at localhost/37495] master.HMaster(439): hbase.rootdir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718, hbase.cluster.distributed=false 2023-02-09 18:00:28,988 INFO [Listener at localhost/37495] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:28,989 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:28,989 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:28,989 INFO [Listener at localhost/37495] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:28,989 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:28,989 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:28,990 INFO [Listener at localhost/37495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:28,991 INFO [Listener at localhost/37495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37375 2023-02-09 18:00:28,991 INFO [Listener at localhost/37495] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 18:00:28,992 DEBUG [Listener at localhost/37495] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 18:00:28,993 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:28,994 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:28,995 INFO [Listener at localhost/37495] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37375 connecting to ZooKeeper ensemble=127.0.0.1:50070 2023-02-09 18:00:28,998 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:373750x0, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:28,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37375-0x1003a3debac0001 connected 2023-02-09 18:00:28,999 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:29,000 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:29,000 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:29,001 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37375 2023-02-09 18:00:29,001 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37375 2023-02-09 18:00:29,001 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37375 2023-02-09 18:00:29,001 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37375 2023-02-09 18:00:29,001 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37375 2023-02-09 18:00:29,012 INFO [Listener at localhost/37495] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:29,012 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:29,012 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:29,013 INFO [Listener at localhost/37495] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:29,013 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:29,013 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:29,013 INFO [Listener at localhost/37495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:29,014 INFO [Listener at localhost/37495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39245 2023-02-09 18:00:29,014 INFO [Listener at localhost/37495] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 18:00:29,015 DEBUG [Listener at localhost/37495] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 18:00:29,016 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:29,017 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:29,018 INFO [Listener at localhost/37495] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39245 connecting to ZooKeeper ensemble=127.0.0.1:50070 2023-02-09 18:00:29,021 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:392450x0, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:29,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39245-0x1003a3debac0002 connected 2023-02-09 18:00:29,022 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:29,022 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:29,023 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:29,023 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39245 2023-02-09 18:00:29,023 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39245 2023-02-09 18:00:29,024 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39245 2023-02-09 18:00:29,026 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39245 2023-02-09 18:00:29,027 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39245 2023-02-09 18:00:29,037 INFO [Listener at localhost/37495] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-09 18:00:29,037 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:29,037 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:29,038 INFO [Listener at localhost/37495] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 18:00:29,038 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 18:00:29,038 INFO [Listener at localhost/37495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 18:00:29,038 INFO [Listener at localhost/37495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 18:00:29,040 INFO [Listener at localhost/37495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38605 2023-02-09 18:00:29,040 INFO [Listener at localhost/37495] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 18:00:29,041 DEBUG [Listener at localhost/37495] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 18:00:29,042 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:29,043 INFO [Listener at localhost/37495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:29,045 INFO [Listener at localhost/37495] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38605 connecting to ZooKeeper ensemble=127.0.0.1:50070 2023-02-09 18:00:29,048 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:386050x0, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 18:00:29,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38605-0x1003a3debac0003 connected 2023-02-09 18:00:29,049 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:29,050 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:29,050 DEBUG [Listener at localhost/37495] zookeeper.ZKUtil(164): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 18:00:29,051 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38605 2023-02-09 18:00:29,052 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38605 2023-02-09 18:00:29,056 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38605 2023-02-09 18:00:29,056 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38605 2023-02-09 18:00:29,056 DEBUG [Listener at localhost/37495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38605 2023-02-09 18:00:29,059 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,060 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-09 18:00:29,061 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,062 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:29,062 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:29,062 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:29,062 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:29,062 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-09 18:00:29,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46519,1675965628928 from backup master directory 2023-02-09 18:00:29,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-09 18:00:29,067 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,067 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:29,067 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-09 18:00:29,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/hbase.id with ID: 0f26474a-cb87-4d14-9626-1c72344c793e 2023-02-09 18:00:29,104 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 18:00:29,108 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,123 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6a5879ad to 127.0.0.1:50070 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:29,127 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57ce96c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:29,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-09 18:00:29,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-09 18:00:29,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:29,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store-tmp 2023-02-09 18:00:29,148 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:29,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-09 18:00:29,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:29,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:29,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-09 18:00:29,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:29,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:29,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1675965629149Disabling compacts and flushes for region at 1675965629149Disabling writes for close at 1675965629149Writing region close event to WAL at 1675965629149Closed at 1675965629149 2023-02-09 18:00:29,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/WALs/jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46519%2C1675965628928, suffix=, logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/WALs/jenkins-hbase4.apache.org,46519,1675965628928, archiveDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/oldWALs, maxLogs=10 2023-02-09 18:00:29,169 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK] 2023-02-09 18:00:29,170 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK] 2023-02-09 18:00:29,171 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK] 2023-02-09 18:00:29,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/WALs/jenkins-hbase4.apache.org,46519,1675965628928/jenkins-hbase4.apache.org%2C46519%2C1675965628928.1675965629153 2023-02-09 18:00:29,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK], DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK], DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK]] 2023-02-09 18:00:29,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-09 18:00:29,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:29,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:29,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:29,179 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:29,181 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-09 18:00:29,181 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-09 18:00:29,182 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,183 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:29,183 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:29,186 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 18:00:29,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 18:00:29,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62006709, jitterRate=-0.07602803409099579}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 18:00:29,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1675965629177Initializing all the Stores at 1675965629178 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675965629178Cleaning up temporary data from old regions at 1675965629184 (+6 ms)Cleaning up detritus from prior splits at 1675965629184Region opened successfully at 1675965629189 (+5 ms) 2023-02-09 18:00:29,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-09 18:00:29,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-09 18:00:29,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-09 18:00:29,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 18:00:29,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 18:00:29,192 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-09 18:00:29,192 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-09 18:00:29,192 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-09 18:00:29,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-09 18:00:29,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-09 18:00:29,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-09 18:00:29,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-09 18:00:29,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-09 18:00:29,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-09 18:00:29,208 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,209 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-09 18:00:29,209 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-09 18:00:29,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-09 18:00:29,212 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:29,212 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:29,212 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:29,212 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:29,212 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,46519,1675965628928, sessionid=0x1003a3debac0000, setting cluster-up flag (Was=false) 2023-02-09 18:00:29,216 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-09 18:00:29,223 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,226 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-09 18:00:29,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:29,232 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/.hbase-snapshot/.tmp 2023-02-09 18:00:29,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:29,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675965659236 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-09 18:00:29,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,237 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-09 18:00:29,238 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-09 18:00:29,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-09 18:00:29,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-09 18:00:29,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-09 18:00:29,239 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-09 18:00:29,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-09 18:00:29,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-09 18:00:29,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1675965629241,5,FailOnTimeoutGroup] 2023-02-09 18:00:29,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1675965629241,5,FailOnTimeoutGroup] 2023-02-09 18:00:29,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-09 18:00:29,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,260 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(952): ClusterId : 0f26474a-cb87-4d14-9626-1c72344c793e 2023-02-09 18:00:29,261 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(952): ClusterId : 0f26474a-cb87-4d14-9626-1c72344c793e 2023-02-09 18:00:29,260 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(952): ClusterId : 0f26474a-cb87-4d14-9626-1c72344c793e 2023-02-09 18:00:29,262 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 18:00:29,262 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 18:00:29,263 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 18:00:29,266 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 18:00:29,267 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 18:00:29,267 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 18:00:29,267 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 18:00:29,267 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 18:00:29,267 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 18:00:29,268 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-09 18:00:29,269 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-09 18:00:29,269 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718 2023-02-09 18:00:29,271 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 18:00:29,272 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 18:00:29,273 DEBUG [RS:2;jenkins-hbase4:38605] zookeeper.ReadOnlyZKClient(139): Connect 0x04bab70e to 127.0.0.1:50070 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:29,273 DEBUG [RS:0;jenkins-hbase4:37375] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd7f62b to 127.0.0.1:50070 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:29,274 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 18:00:29,280 DEBUG [RS:1;jenkins-hbase4:39245] zookeeper.ReadOnlyZKClient(139): Connect 0x07588199 to 127.0.0.1:50070 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:29,288 DEBUG [RS:2;jenkins-hbase4:38605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@853341, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:29,289 DEBUG [RS:2;jenkins-hbase4:38605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4587bd35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:29,290 DEBUG [RS:0;jenkins-hbase4:37375] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32101fc1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:29,290 DEBUG [RS:0;jenkins-hbase4:37375] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@362f9d5b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:29,297 DEBUG [RS:1;jenkins-hbase4:39245] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@da0eae9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:29,298 DEBUG [RS:1;jenkins-hbase4:39245] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48a26539, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:29,304 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:38605 2023-02-09 18:00:29,304 INFO [RS:2;jenkins-hbase4:38605] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 18:00:29,304 INFO [RS:2;jenkins-hbase4:38605] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 18:00:29,304 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 18:00:29,305 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46519,1675965628928 with isa=jenkins-hbase4.apache.org/172.31.14.131:38605, startcode=1675965629037 2023-02-09 18:00:29,305 DEBUG [RS:2;jenkins-hbase4:38605] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 18:00:29,307 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:37375 2023-02-09 18:00:29,307 INFO [RS:0;jenkins-hbase4:37375] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 18:00:29,307 INFO [RS:0;jenkins-hbase4:37375] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 18:00:29,307 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 18:00:29,309 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46519,1675965628928 with isa=jenkins-hbase4.apache.org/172.31.14.131:37375, startcode=1675965628988 2023-02-09 18:00:29,310 DEBUG [RS:0;jenkins-hbase4:37375] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 18:00:29,310 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:29,310 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54035, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 18:00:29,311 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39245 2023-02-09 18:00:29,311 INFO [RS:1;jenkins-hbase4:39245] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 18:00:29,311 INFO [RS:1;jenkins-hbase4:39245] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 18:00:29,311 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 18:00:29,312 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46519,1675965628928 with isa=jenkins-hbase4.apache.org/172.31.14.131:39245, startcode=1675965629012 2023-02-09 18:00:29,312 DEBUG [RS:1;jenkins-hbase4:39245] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 18:00:29,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46519] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,317 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-09 18:00:29,318 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718 2023-02-09 18:00:29,318 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41311, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 18:00:29,318 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35575 2023-02-09 18:00:29,318 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52683, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 18:00:29,318 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 18:00:29,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46519] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,318 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46519] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,319 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718 2023-02-09 18:00:29,319 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35575 2023-02-09 18:00:29,319 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 18:00:29,320 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/info 2023-02-09 18:00:29,321 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-09 18:00:29,321 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:29,321 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718 2023-02-09 18:00:29,322 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,322 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35575 2023-02-09 18:00:29,322 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 18:00:29,322 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-09 18:00:29,324 DEBUG [RS:2;jenkins-hbase4:38605] zookeeper.ZKUtil(162): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,324 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/rep_barrier 2023-02-09 18:00:29,324 WARN [RS:2;jenkins-hbase4:38605] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:29,325 INFO [RS:2;jenkins-hbase4:38605] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:29,325 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38605,1675965629037] 2023-02-09 18:00:29,325 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37375,1675965628988] 2023-02-09 18:00:29,325 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,325 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-09 18:00:29,325 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:29,326 DEBUG [RS:0;jenkins-hbase4:37375] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,328 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,328 WARN [RS:0;jenkins-hbase4:37375] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:29,329 DEBUG [RS:1;jenkins-hbase4:39245] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,329 INFO [RS:0;jenkins-hbase4:37375] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:29,329 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39245,1675965629012] 2023-02-09 18:00:29,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-09 18:00:29,329 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,329 WARN [RS:1;jenkins-hbase4:39245] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 18:00:29,330 INFO [RS:1;jenkins-hbase4:39245] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:29,330 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,334 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/table 2023-02-09 18:00:29,334 DEBUG [RS:2;jenkins-hbase4:38605] zookeeper.ZKUtil(162): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,335 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-09 18:00:29,336 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,337 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740 2023-02-09 18:00:29,337 DEBUG [RS:2;jenkins-hbase4:38605] zookeeper.ZKUtil(162): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,338 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740 2023-02-09 18:00:29,338 DEBUG [RS:2;jenkins-hbase4:38605] zookeeper.ZKUtil(162): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,338 DEBUG [RS:1;jenkins-hbase4:39245] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,339 DEBUG [RS:0;jenkins-hbase4:37375] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,339 DEBUG [RS:1;jenkins-hbase4:39245] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,339 DEBUG [RS:0;jenkins-hbase4:37375] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,340 DEBUG [RS:1;jenkins-hbase4:39245] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,340 DEBUG [RS:2;jenkins-hbase4:38605] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 18:00:29,340 INFO [RS:2;jenkins-hbase4:38605] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 18:00:29,340 DEBUG [RS:0;jenkins-hbase4:37375] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,341 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 18:00:29,341 DEBUG [RS:1;jenkins-hbase4:39245] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 18:00:29,343 INFO [RS:0;jenkins-hbase4:37375] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 18:00:29,343 INFO [RS:1;jenkins-hbase4:39245] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 18:00:29,344 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-09 18:00:29,344 INFO [RS:2;jenkins-hbase4:38605] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 18:00:29,346 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-09 18:00:29,348 INFO [RS:2;jenkins-hbase4:38605] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 18:00:29,348 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,351 INFO [RS:1;jenkins-hbase4:39245] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 18:00:29,351 INFO [RS:0;jenkins-hbase4:37375] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 18:00:29,351 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 18:00:29,354 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,354 INFO [RS:0;jenkins-hbase4:37375] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 18:00:29,354 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,354 INFO [RS:1;jenkins-hbase4:39245] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 18:00:29,354 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,354 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,354 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,354 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:29,355 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,355 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,355 DEBUG [RS:2;jenkins-hbase4:38605] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,356 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 18:00:29,356 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71571035, jitterRate=0.06649152934551239}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-09 18:00:29,356 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1675965629310Initializing all the Stores at 1675965629312 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965629312Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675965629316 (+4 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965629316Cleaning up temporary data from old regions at 1675965629340 (+24 ms)Cleaning up detritus from prior splits at 1675965629342 (+2 ms)Region opened successfully at 1675965629356 (+14 ms) 2023-02-09 18:00:29,357 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-09 18:00:29,357 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-09 18:00:29,357 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-09 18:00:29,357 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-09 18:00:29,357 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-09 18:00:29,361 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,362 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,362 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,362 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-09 18:00:29,362 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1675965629357Disabling compacts and flushes for region at 1675965629357Disabling writes for close at 1675965629357Writing region close event to WAL at 1675965629362 (+5 ms)Closed at 1675965629362 2023-02-09 18:00:29,364 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,364 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-09 18:00:29,364 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-09 18:00:29,364 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,364 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,364 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,364 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-09 18:00:29,364 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,364 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,364 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:29,365 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,365 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,365 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,365 DEBUG [RS:0;jenkins-hbase4:37375] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,374 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,375 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,376 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-09 18:00:29,376 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,376 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,380 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,381 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,381 DEBUG [RS:1;jenkins-hbase4:39245] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-09 18:00:29,382 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-09 18:00:29,386 INFO [RS:2;jenkins-hbase4:38605] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 18:00:29,386 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38605,1675965629037-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,389 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,389 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,389 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,399 INFO [RS:0;jenkins-hbase4:37375] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 18:00:29,399 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37375,1675965628988-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,402 INFO [RS:2;jenkins-hbase4:38605] regionserver.Replication(203): jenkins-hbase4.apache.org,38605,1675965629037 started 2023-02-09 18:00:29,402 INFO [RS:1;jenkins-hbase4:39245] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 18:00:29,402 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38605,1675965629037, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38605, sessionid=0x1003a3debac0003 2023-02-09 18:00:29,403 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39245,1675965629012-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,403 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 18:00:29,403 DEBUG [RS:2;jenkins-hbase4:38605] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,403 DEBUG [RS:2;jenkins-hbase4:38605] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38605,1675965629037' 2023-02-09 18:00:29,403 DEBUG [RS:2;jenkins-hbase4:38605] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 18:00:29,403 DEBUG [RS:2;jenkins-hbase4:38605] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 18:00:29,404 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 18:00:29,404 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 18:00:29,404 DEBUG [RS:2;jenkins-hbase4:38605] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:29,404 DEBUG [RS:2;jenkins-hbase4:38605] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38605,1675965629037' 2023-02-09 18:00:29,404 DEBUG [RS:2;jenkins-hbase4:38605] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 18:00:29,405 DEBUG [RS:2;jenkins-hbase4:38605] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 18:00:29,406 DEBUG [RS:2;jenkins-hbase4:38605] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 18:00:29,406 INFO [RS:2;jenkins-hbase4:38605] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 18:00:29,406 INFO [RS:2;jenkins-hbase4:38605] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 18:00:29,414 INFO [RS:1;jenkins-hbase4:39245] regionserver.Replication(203): jenkins-hbase4.apache.org,39245,1675965629012 started 2023-02-09 18:00:29,414 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39245,1675965629012, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39245, sessionid=0x1003a3debac0002 2023-02-09 18:00:29,414 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 18:00:29,414 DEBUG [RS:1;jenkins-hbase4:39245] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,414 DEBUG [RS:1;jenkins-hbase4:39245] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39245,1675965629012' 2023-02-09 18:00:29,414 DEBUG [RS:1;jenkins-hbase4:39245] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 18:00:29,415 DEBUG [RS:1;jenkins-hbase4:39245] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 18:00:29,415 INFO [RS:0;jenkins-hbase4:37375] regionserver.Replication(203): jenkins-hbase4.apache.org,37375,1675965628988 started 2023-02-09 18:00:29,415 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37375,1675965628988, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37375, sessionid=0x1003a3debac0001 2023-02-09 18:00:29,415 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 18:00:29,415 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 18:00:29,415 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 18:00:29,416 DEBUG [RS:0;jenkins-hbase4:37375] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,416 DEBUG [RS:1;jenkins-hbase4:39245] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:29,416 DEBUG [RS:1;jenkins-hbase4:39245] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39245,1675965629012' 2023-02-09 18:00:29,416 DEBUG [RS:1;jenkins-hbase4:39245] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 18:00:29,416 DEBUG [RS:0;jenkins-hbase4:37375] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37375,1675965628988' 2023-02-09 18:00:29,416 DEBUG [RS:0;jenkins-hbase4:37375] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 18:00:29,416 DEBUG [RS:1;jenkins-hbase4:39245] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 18:00:29,416 DEBUG [RS:0;jenkins-hbase4:37375] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 18:00:29,417 DEBUG [RS:1;jenkins-hbase4:39245] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 18:00:29,417 INFO [RS:1;jenkins-hbase4:39245] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 18:00:29,417 INFO [RS:1;jenkins-hbase4:39245] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 18:00:29,417 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 18:00:29,417 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 18:00:29,417 DEBUG [RS:0;jenkins-hbase4:37375] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,417 DEBUG [RS:0;jenkins-hbase4:37375] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37375,1675965628988' 2023-02-09 18:00:29,417 DEBUG [RS:0;jenkins-hbase4:37375] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 18:00:29,417 DEBUG [RS:0;jenkins-hbase4:37375] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 18:00:29,418 DEBUG [RS:0;jenkins-hbase4:37375] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 18:00:29,418 INFO [RS:0;jenkins-hbase4:37375] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 18:00:29,418 INFO [RS:0;jenkins-hbase4:37375] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 18:00:29,508 INFO [RS:2;jenkins-hbase4:38605] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38605%2C1675965629037, suffix=, logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,38605,1675965629037, archiveDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs, maxLogs=32 2023-02-09 18:00:29,519 INFO [RS:1;jenkins-hbase4:39245] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39245%2C1675965629012, suffix=, logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,39245,1675965629012, archiveDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs, maxLogs=32 2023-02-09 18:00:29,520 INFO [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37375%2C1675965628988, suffix=, logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,37375,1675965628988, archiveDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs, maxLogs=32 2023-02-09 18:00:29,531 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK] 2023-02-09 18:00:29,532 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK] 2023-02-09 18:00:29,532 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK] 2023-02-09 18:00:29,538 DEBUG [jenkins-hbase4:46519] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-09 18:00:29,538 DEBUG [jenkins-hbase4:46519] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-09 18:00:29,540 INFO [RS:2;jenkins-hbase4:38605] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,38605,1675965629037/jenkins-hbase4.apache.org%2C38605%2C1675965629037.1675965629510 2023-02-09 18:00:29,540 DEBUG [RS:2;jenkins-hbase4:38605] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK], DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK], DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK]] 2023-02-09 18:00:29,541 DEBUG [jenkins-hbase4:46519] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 18:00:29,541 DEBUG [jenkins-hbase4:46519] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 18:00:29,541 DEBUG [jenkins-hbase4:46519] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 18:00:29,541 DEBUG [jenkins-hbase4:46519] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 18:00:29,544 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37375,1675965628988, state=OPENING 2023-02-09 18:00:29,550 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-09 18:00:29,551 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:29,552 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-09 18:00:29,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37375,1675965628988}] 2023-02-09 18:00:29,558 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK] 2023-02-09 18:00:29,558 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK] 2023-02-09 18:00:29,560 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK] 2023-02-09 18:00:29,560 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK] 2023-02-09 18:00:29,561 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK] 2023-02-09 18:00:29,561 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK] 2023-02-09 18:00:29,566 INFO [RS:1;jenkins-hbase4:39245] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,39245,1675965629012/jenkins-hbase4.apache.org%2C39245%2C1675965629012.1675965629524 2023-02-09 18:00:29,566 DEBUG [RS:1;jenkins-hbase4:39245] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK], DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK], DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK]] 2023-02-09 18:00:29,570 INFO [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,37375,1675965628988/jenkins-hbase4.apache.org%2C37375%2C1675965628988.1675965629524 2023-02-09 18:00:29,570 DEBUG [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK], DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK], DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK]] 2023-02-09 18:00:29,717 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:29,717 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 18:00:29,719 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40308, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 18:00:29,724 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-09 18:00:29,724 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 18:00:29,726 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37375%2C1675965628988.meta, suffix=.meta, logDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,37375,1675965628988, archiveDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs, maxLogs=32 2023-02-09 18:00:29,740 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK] 2023-02-09 18:00:29,741 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK] 2023-02-09 18:00:29,741 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK] 2023-02-09 18:00:29,744 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/WALs/jenkins-hbase4.apache.org,37375,1675965628988/jenkins-hbase4.apache.org%2C37375%2C1675965628988.meta.1675965629726.meta 2023-02-09 18:00:29,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46031,DS-fe179a80-d98a-414a-ae37-3904651655d9,DISK], DatanodeInfoWithStorage[127.0.0.1:46517,DS-9dea9131-f341-4fcd-b6fc-c2343120c3a4,DISK], DatanodeInfoWithStorage[127.0.0.1:34185,DS-39d61341-2c1f-4579-8962-4d63660d2dae,DISK]] 2023-02-09 18:00:29,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-09 18:00:29,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-09 18:00:29,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-09 18:00:29,745 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-09 18:00:29,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-09 18:00:29,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:29,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-09 18:00:29,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-09 18:00:29,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-09 18:00:29,748 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/info 2023-02-09 18:00:29,748 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/info 2023-02-09 18:00:29,749 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-09 18:00:29,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-09 18:00:29,750 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/rep_barrier 2023-02-09 18:00:29,750 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/rep_barrier 2023-02-09 18:00:29,751 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-09 18:00:29,751 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,751 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-09 18:00:29,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/table 2023-02-09 18:00:29,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/table 2023-02-09 18:00:29,753 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-09 18:00:29,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:29,754 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740 2023-02-09 18:00:29,756 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740 2023-02-09 18:00:29,760 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-09 18:00:29,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-09 18:00:29,763 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72826888, jitterRate=0.08520519733428955}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-09 18:00:29,763 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1675965629745Writing region info on filesystem at 1675965629745Initializing all the Stores at 1675965629747 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965629747Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675965629747Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965629747Cleaning up temporary data from old regions at 1675965629757 (+10 ms)Cleaning up detritus from prior splits at 1675965629759 (+2 ms)Running coprocessor post-open hooks at 1675965629763 (+4 ms)Region opened successfully at 1675965629763 2023-02-09 18:00:29,764 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1675965629717 2023-02-09 18:00:29,769 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-09 18:00:29,769 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-09 18:00:29,770 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37375,1675965628988, state=OPEN 2023-02-09 18:00:29,772 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-09 18:00:29,772 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-09 18:00:29,777 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-09 18:00:29,777 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37375,1675965628988 in 220 msec 2023-02-09 18:00:29,780 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-09 18:00:29,780 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 413 msec 2023-02-09 18:00:29,783 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 548 msec 2023-02-09 18:00:29,783 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1675965629061, completionTime=-1 2023-02-09 18:00:29,783 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-09 18:00:29,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 18:00:29,786 DEBUG [hconnection-0x3c65ed3e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 18:00:29,788 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40314, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 18:00:29,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 18:00:29,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675965689789 2023-02-09 18:00:29,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675965749789 2023-02-09 18:00:29,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 6 msec 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46519,1675965628928-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46519,1675965628928-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46519,1675965628928-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46519, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-09 18:00:29,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-09 18:00:29,798 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-09 18:00:29,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-09 18:00:29,800 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 18:00:29,801 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 18:00:29,803 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/.tmp/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:29,804 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/.tmp/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48 empty. 2023-02-09 18:00:29,804 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/.tmp/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:29,804 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-09 18:00:29,820 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-09 18:00:29,821 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => a91b79ce7a1bbd77ef5363f971511c48, NAME => 'hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/.tmp 2023-02-09 18:00:29,836 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:29,836 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing a91b79ce7a1bbd77ef5363f971511c48, disabling compactions & flushes 2023-02-09 18:00:29,836 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:29,836 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:29,836 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. after waiting 0 ms 2023-02-09 18:00:29,836 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:29,836 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:29,836 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for a91b79ce7a1bbd77ef5363f971511c48: Waiting for close lock at 1675965629836Disabling compacts and flushes for region at 1675965629836Disabling writes for close at 1675965629836Writing region close event to WAL at 1675965629836Closed at 1675965629836 2023-02-09 18:00:29,839 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 18:00:29,841 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675965629841"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675965629841"}]},"ts":"1675965629841"} 2023-02-09 18:00:29,844 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 18:00:29,846 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 18:00:29,846 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675965629846"}]},"ts":"1675965629846"} 2023-02-09 18:00:29,848 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-09 18:00:29,851 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-09 18:00:29,852 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 18:00:29,852 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 18:00:29,852 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 18:00:29,852 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 18:00:29,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a91b79ce7a1bbd77ef5363f971511c48, ASSIGN}] 2023-02-09 18:00:29,854 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a91b79ce7a1bbd77ef5363f971511c48, ASSIGN 2023-02-09 18:00:29,855 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a91b79ce7a1bbd77ef5363f971511c48, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39245,1675965629012; forceNewPlan=false, retain=false 2023-02-09 18:00:30,006 INFO [jenkins-hbase4:46519] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 18:00:30,007 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a91b79ce7a1bbd77ef5363f971511c48, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,007 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675965630007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675965630007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675965630007"}]},"ts":"1675965630007"} 2023-02-09 18:00:30,010 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a91b79ce7a1bbd77ef5363f971511c48, server=jenkins-hbase4.apache.org,39245,1675965629012}] 2023-02-09 18:00:30,164 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,164 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 18:00:30,166 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35296, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 18:00:30,171 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a91b79ce7a1bbd77ef5363f971511c48, NAME => 'hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.', STARTKEY => '', ENDKEY => ''} 2023-02-09 18:00:30,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 18:00:30,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,173 INFO [StoreOpener-a91b79ce7a1bbd77ef5363f971511c48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,175 DEBUG [StoreOpener-a91b79ce7a1bbd77ef5363f971511c48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/info 2023-02-09 18:00:30,175 DEBUG [StoreOpener-a91b79ce7a1bbd77ef5363f971511c48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/info 2023-02-09 18:00:30,176 INFO [StoreOpener-a91b79ce7a1bbd77ef5363f971511c48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a91b79ce7a1bbd77ef5363f971511c48 columnFamilyName info 2023-02-09 18:00:30,177 INFO [StoreOpener-a91b79ce7a1bbd77ef5363f971511c48-1] regionserver.HStore(310): Store=a91b79ce7a1bbd77ef5363f971511c48/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 18:00:30,178 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,179 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,183 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,186 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 18:00:30,186 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a91b79ce7a1bbd77ef5363f971511c48; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67486566, jitterRate=0.0056281983852386475}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 18:00:30,186 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a91b79ce7a1bbd77ef5363f971511c48: Running coprocessor pre-open hook at 1675965630172Writing region info on filesystem at 1675965630172Initializing all the Stores at 1675965630173 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675965630173Cleaning up temporary data from old regions at 1675965630180 (+7 ms)Cleaning up detritus from prior splits at 1675965630180Running coprocessor post-open hooks at 1675965630186 (+6 ms)Region opened successfully at 1675965630186 2023-02-09 18:00:30,188 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48., pid=6, masterSystemTime=1675965630164 2023-02-09 18:00:30,193 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,194 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,195 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a91b79ce7a1bbd77ef5363f971511c48, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,196 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675965630195"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675965630195"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675965630195"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675965630195"}]},"ts":"1675965630195"} 2023-02-09 18:00:30,201 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-09 18:00:30,201 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a91b79ce7a1bbd77ef5363f971511c48, server=jenkins-hbase4.apache.org,39245,1675965629012 in 188 msec 2023-02-09 18:00:30,204 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-09 18:00:30,205 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a91b79ce7a1bbd77ef5363f971511c48, ASSIGN in 349 msec 2023-02-09 18:00:30,206 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 18:00:30,206 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675965630206"}]},"ts":"1675965630206"} 2023-02-09 18:00:30,208 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-09 18:00:30,210 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 18:00:30,212 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 414 msec 2023-02-09 18:00:30,300 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-09 18:00:30,302 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-09 18:00:30,302 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:30,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 18:00:30,307 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35298, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 18:00:30,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-09 18:00:30,321 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-09 18:00:30,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 15 msec 2023-02-09 18:00:30,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-09 18:00:30,341 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-09 18:00:30,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-09 18:00:30,356 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-09 18:00:30,360 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-09 18:00:30,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.293sec 2023-02-09 18:00:30,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-09 18:00:30,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-09 18:00:30,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-09 18:00:30,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46519,1675965628928-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-09 18:00:30,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46519,1675965628928-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-09 18:00:30,361 DEBUG [Listener at localhost/37495] zookeeper.ReadOnlyZKClient(139): Connect 0x5c7748f1 to 127.0.0.1:50070 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:30,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-09 18:00:30,366 DEBUG [Listener at localhost/37495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36ae13be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:30,367 DEBUG [hconnection-0x7a5d71cf-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 18:00:30,369 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40320, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 18:00:30,371 INFO [Listener at localhost/37495] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:30,371 DEBUG [Listener at localhost/37495] zookeeper.ReadOnlyZKClient(139): Connect 0x661a42c5 to 127.0.0.1:50070 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 18:00:30,377 DEBUG [ReadOnlyZKClient-127.0.0.1:50070@0x661a42c5] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46ac950f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 18:00:30,378 DEBUG [Listener at localhost/37495] client.ConnectionUtils(586): Start fetching master stub from registry 2023-02-09 18:00:30,379 DEBUG [ReadOnlyZKClient-127.0.0.1:50070@0x661a42c5] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:30,379 DEBUG [ReadOnlyZKClient-127.0.0.1:50070@0x661a42c5] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@42931902 2023-02-09 18:00:30,383 DEBUG [ReadOnlyZKClient-127.0.0.1:50070@0x661a42c5] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-09 18:00:30,385 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41856, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-09 18:00:30,385 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46519] master.MasterRpcServices(1560): Client=jenkins//172.31.14.131 shutdown 2023-02-09 18:00:30,385 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46519] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:30,387 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:30,387 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:30,387 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:30,387 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:30,387 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 18:00:30,387 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46519] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 18:00:30,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:30,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:30,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:30,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 18:00:30,389 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46519] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a5879ad to 127.0.0.1:50070 2023-02-09 18:00:30,389 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46519] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,452 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38605,1675965629037' ***** 2023-02-09 18:00:30,452 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-09 18:00:30,453 INFO [RS:2;jenkins-hbase4:38605] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 18:00:30,453 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 18:00:30,453 INFO [RS:2;jenkins-hbase4:38605] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 18:00:30,454 INFO [RS:2;jenkins-hbase4:38605] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 18:00:30,454 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:30,454 DEBUG [RS:2;jenkins-hbase4:38605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04bab70e to 127.0.0.1:50070 2023-02-09 18:00:30,454 DEBUG [RS:2;jenkins-hbase4:38605] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,454 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38605,1675965629037; all regions closed. 2023-02-09 18:00:30,456 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-09 18:00:30,456 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-09 18:00:30,462 DEBUG [RS:2;jenkins-hbase4:38605] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs 2023-02-09 18:00:30,462 INFO [RS:2;jenkins-hbase4:38605] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38605%2C1675965629037:(num 1675965629510) 2023-02-09 18:00:30,462 DEBUG [RS:2;jenkins-hbase4:38605] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,462 INFO [RS:2;jenkins-hbase4:38605] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:30,462 INFO [RS:2;jenkins-hbase4:38605] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-09 18:00:30,462 INFO [RS:2;jenkins-hbase4:38605] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 18:00:30,462 INFO [RS:2;jenkins-hbase4:38605] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 18:00:30,462 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:30,462 INFO [RS:2;jenkins-hbase4:38605] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 18:00:30,463 INFO [RS:2;jenkins-hbase4:38605] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38605 2023-02-09 18:00:30,465 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:30,467 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:30,467 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:30,467 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,467 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 2023-02-09 18:00:30,467 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,467 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,468 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,468 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1f3096e8 rejected from java.util.concurrent.ThreadPoolExecutor@1b0f11d6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,469 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38605,1675965629037] 2023-02-09 18:00:30,470 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38605,1675965629037; numProcessing=1 2023-02-09 18:00:30,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,473 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38605,1675965629037 already deleted, retry=false 2023-02-09 18:00:30,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,473 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38605,1675965629037 expired; onlineServers=2 2023-02-09 18:00:30,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,473 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 znode expired, triggering replicatorRemoved event 2023-02-09 18:00:30,474 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,38605,1675965629037 znode expired, triggering replicatorRemoved event 2023-02-09 18:00:30,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,480 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1065): Closing user regions 2023-02-09 18:00:30,480 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(3304): Received CLOSE for a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a91b79ce7a1bbd77ef5363f971511c48, disabling compactions & flushes 2023-02-09 18:00:30,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. after waiting 0 ms 2023-02-09 18:00:30,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a91b79ce7a1bbd77ef5363f971511c48 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-09 18:00:30,484 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1065): Closing user regions 2023-02-09 18:00:30,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/.tmp/info/f38126e9663a4dfaa150b4044bbd3256 2023-02-09 18:00:30,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/.tmp/info/f38126e9663a4dfaa150b4044bbd3256 as hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/info/f38126e9663a4dfaa150b4044bbd3256 2023-02-09 18:00:30,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/info/f38126e9663a4dfaa150b4044bbd3256, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 18:00:30,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a91b79ce7a1bbd77ef5363f971511c48 in 42ms, sequenceid=6, compaction requested=false 2023-02-09 18:00:30,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/namespace/a91b79ce7a1bbd77ef5363f971511c48/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-09 18:00:30,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a91b79ce7a1bbd77ef5363f971511c48: Waiting for close lock at 1675965630480Running coprocessor pre-close hooks at 1675965630480Disabling compacts and flushes for region at 1675965630480Disabling writes for close at 1675965630481 (+1 ms)Obtaining lock to block concurrent updates at 1675965630481Preparing flush snapshotting stores in a91b79ce7a1bbd77ef5363f971511c48 at 1675965630481Finished memstore snapshotting hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1675965630481Flushing stores of hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. at 1675965630482 (+1 ms)Flushing a91b79ce7a1bbd77ef5363f971511c48/info: creating writer at 1675965630482Flushing a91b79ce7a1bbd77ef5363f971511c48/info: appending metadata at 1675965630486 (+4 ms)Flushing a91b79ce7a1bbd77ef5363f971511c48/info: closing flushed file at 1675965630486Flushing a91b79ce7a1bbd77ef5363f971511c48/info: reopening flushed file at 1675965630514 (+28 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a91b79ce7a1bbd77ef5363f971511c48 in 42ms, sequenceid=6, compaction requested=false at 1675965630523 (+9 ms)Writing region close event to WAL at 1675965630526 (+3 ms)Running coprocessor post-close hooks at 1675965630532 (+6 ms)Closed at 1675965630532 2023-02-09 18:00:30,533 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46519] assignment.AssignmentManager(1094): RegionServer CLOSED a91b79ce7a1bbd77ef5363f971511c48 2023-02-09 18:00:30,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1675965629797.a91b79ce7a1bbd77ef5363f971511c48. 2023-02-09 18:00:30,582 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39245,1675965629012' ***** 2023-02-09 18:00:30,582 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-09 18:00:30,584 INFO [RS:1;jenkins-hbase4:39245] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 18:00:30,584 INFO [RS:1;jenkins-hbase4:39245] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 18:00:30,584 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 18:00:30,584 INFO [RS:1;jenkins-hbase4:39245] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 18:00:30,585 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,585 DEBUG [RS:1;jenkins-hbase4:39245] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07588199 to 127.0.0.1:50070 2023-02-09 18:00:30,585 DEBUG [RS:1;jenkins-hbase4:39245] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,585 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39245,1675965629012; all regions closed. 2023-02-09 18:00:30,586 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1084): Waiting on 1588230740 2023-02-09 18:00:30,591 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:30,592 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:30,592 INFO [RS:2;jenkins-hbase4:38605] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38605,1675965629037; zookeeper connection closed. 2023-02-09 18:00:30,592 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7405fffe rejected from java.util.concurrent.ThreadPoolExecutor@1b0f11d6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,593 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:38605-0x1003a3debac0003, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:30,593 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2dda0f8d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2dda0f8d 2023-02-09 18:00:30,593 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@cfb9dd4 rejected from java.util.concurrent.ThreadPoolExecutor@1b0f11d6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,595 DEBUG [RS:1;jenkins-hbase4:39245] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs 2023-02-09 18:00:30,595 INFO [RS:1;jenkins-hbase4:39245] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39245%2C1675965629012:(num 1675965629524) 2023-02-09 18:00:30,595 DEBUG [RS:1;jenkins-hbase4:39245] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,595 INFO [RS:1;jenkins-hbase4:39245] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:30,595 INFO [RS:1;jenkins-hbase4:39245] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-09 18:00:30,595 INFO [RS:1;jenkins-hbase4:39245] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 18:00:30,595 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:30,595 INFO [RS:1;jenkins-hbase4:39245] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 18:00:30,595 INFO [RS:1;jenkins-hbase4:39245] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 18:00:30,596 INFO [RS:1;jenkins-hbase4:39245] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39245 2023-02-09 18:00:30,600 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,600 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 2023-02-09 18:00:30,600 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,600 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,600 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@378a2e49 rejected from java.util.concurrent.ThreadPoolExecutor@33b5ac60[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,600 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,602 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39245,1675965629012] 2023-02-09 18:00:30,602 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39245,1675965629012; numProcessing=2 2023-02-09 18:00:30,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,602 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39245,1675965629012 znode expired, triggering replicatorRemoved event 2023-02-09 18:00:30,604 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39245,1675965629012 already deleted, retry=false 2023-02-09 18:00:30,604 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39245,1675965629012 expired; onlineServers=1 2023-02-09 18:00:30,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,689 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37375,1675965628988' ***** 2023-02-09 18:00:30,689 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(2310): STOPPED: Stopped; only catalog regions remaining online 2023-02-09 18:00:30,690 INFO [RS:0;jenkins-hbase4:37375] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 18:00:30,690 INFO [RS:0;jenkins-hbase4:37375] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 18:00:30,690 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 18:00:30,690 INFO [RS:0;jenkins-hbase4:37375] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 18:00:30,690 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,691 DEBUG [RS:0;jenkins-hbase4:37375] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd7f62b to 127.0.0.1:50070 2023-02-09 18:00:30,691 DEBUG [RS:0;jenkins-hbase4:37375] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,691 INFO [RS:0;jenkins-hbase4:37375] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 18:00:30,691 INFO [RS:0;jenkins-hbase4:37375] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 18:00:30,691 INFO [RS:0;jenkins-hbase4:37375] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 18:00:30,691 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:30,691 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-09 18:00:30,692 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-09 18:00:30,692 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-09 18:00:30,692 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-09 18:00:30,692 DEBUG [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-09 18:00:30,692 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-09 18:00:30,692 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-09 18:00:30,692 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-09 18:00:30,692 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-09 18:00:30,692 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-02-09 18:00:30,710 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/.tmp/info/7f8b9af480ba466a969e111c9b9a9ab7 2023-02-09 18:00:30,730 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/.tmp/table/e02825248bb54e35862f5b5323dcecf7 2023-02-09 18:00:30,738 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/.tmp/info/7f8b9af480ba466a969e111c9b9a9ab7 as hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/info/7f8b9af480ba466a969e111c9b9a9ab7 2023-02-09 18:00:30,744 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/info/7f8b9af480ba466a969e111c9b9a9ab7, entries=10, sequenceid=9, filesize=5.9 K 2023-02-09 18:00:30,746 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/.tmp/table/e02825248bb54e35862f5b5323dcecf7 as hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/table/e02825248bb54e35862f5b5323dcecf7 2023-02-09 18:00:30,754 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/table/e02825248bb54e35862f5b5323dcecf7, entries=2, sequenceid=9, filesize=4.7 K 2023-02-09 18:00:30,755 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 63ms, sequenceid=9, compaction requested=false 2023-02-09 18:00:30,761 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-02-09 18:00:30,762 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-09 18:00:30,762 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-09 18:00:30,762 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1675965630692Running coprocessor pre-close hooks at 1675965630692Disabling compacts and flushes for region at 1675965630692Disabling writes for close at 1675965630692Obtaining lock to block concurrent updates at 1675965630693 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1675965630693Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1290, getHeapSize=2912, getOffHeapSize=0, getCellsCount=12 at 1675965630693Flushing stores of hbase:meta,,1.1588230740 at 1675965630694 (+1 ms)Flushing 1588230740/info: creating writer at 1675965630694Flushing 1588230740/info: appending metadata at 1675965630698 (+4 ms)Flushing 1588230740/info: closing flushed file at 1675965630698Flushing 1588230740/table: creating writer at 1675965630717 (+19 ms)Flushing 1588230740/table: appending metadata at 1675965630720 (+3 ms)Flushing 1588230740/table: closing flushed file at 1675965630720Flushing 1588230740/info: reopening flushed file at 1675965630739 (+19 ms)Flushing 1588230740/table: reopening flushed file at 1675965630747 (+8 ms)Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 63ms, sequenceid=9, compaction requested=false at 1675965630755 (+8 ms)Writing region close event to WAL at 1675965630758 (+3 ms)Running coprocessor post-close hooks at 1675965630762 (+4 ms)Closed at 1675965630762 2023-02-09 18:00:30,762 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-09 18:00:30,793 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:30,793 INFO [RS:1;jenkins-hbase4:39245] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39245,1675965629012; zookeeper connection closed. 2023-02-09 18:00:30,793 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@211518f3 rejected from java.util.concurrent.ThreadPoolExecutor@33b5ac60[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,794 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@54abc339] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@54abc339 2023-02-09 18:00:30,794 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:39245-0x1003a3debac0002, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:30,794 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@59c3f0c3 rejected from java.util.concurrent.ThreadPoolExecutor@33b5ac60[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,892 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37375,1675965628988; all regions closed. 2023-02-09 18:00:30,899 DEBUG [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs 2023-02-09 18:00:30,899 INFO [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37375%2C1675965628988.meta:.meta(num 1675965629726) 2023-02-09 18:00:30,905 DEBUG [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/oldWALs 2023-02-09 18:00:30,905 INFO [RS:0;jenkins-hbase4:37375] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37375%2C1675965628988:(num 1675965629524) 2023-02-09 18:00:30,905 DEBUG [RS:0;jenkins-hbase4:37375] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,905 INFO [RS:0;jenkins-hbase4:37375] regionserver.LeaseManager(133): Closed leases 2023-02-09 18:00:30,905 INFO [RS:0;jenkins-hbase4:37375] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-09 18:00:30,905 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:30,906 INFO [RS:0;jenkins-hbase4:37375] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37375 2023-02-09 18:00:30,910 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37375,1675965628988 2023-02-09 18:00:30,910 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,910 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@752393ed rejected from java.util.concurrent.ThreadPoolExecutor@711c0991[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,910 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 18:00:30,910 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6cb516de rejected from java.util.concurrent.ThreadPoolExecutor@711c0991[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:30,912 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37375,1675965628988] 2023-02-09 18:00:30,912 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37375,1675965628988; numProcessing=3 2023-02-09 18:00:30,914 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37375,1675965628988 already deleted, retry=false 2023-02-09 18:00:30,914 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37375,1675965628988 expired; onlineServers=0 2023-02-09 18:00:30,914 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46519,1675965628928' ***** 2023-02-09 18:00:30,914 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-09 18:00:30,915 DEBUG [M:0;jenkins-hbase4:46519] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33f4b962, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-09 18:00:30,915 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46519,1675965628928 2023-02-09 18:00:30,915 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46519,1675965628928; all regions closed. 2023-02-09 18:00:30,915 DEBUG [M:0;jenkins-hbase4:46519] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,915 DEBUG [M:0;jenkins-hbase4:46519] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-09 18:00:30,915 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-09 18:00:30,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1675965629241] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1675965629241,5,FailOnTimeoutGroup] 2023-02-09 18:00:30,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1675965629241] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1675965629241,5,FailOnTimeoutGroup] 2023-02-09 18:00:30,915 DEBUG [M:0;jenkins-hbase4:46519] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-09 18:00:30,917 INFO [M:0;jenkins-hbase4:46519] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-09 18:00:30,917 INFO [M:0;jenkins-hbase4:46519] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-09 18:00:30,917 INFO [M:0;jenkins-hbase4:46519] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-09 18:00:30,917 DEBUG [M:0;jenkins-hbase4:46519] master.HMaster(1502): Stopping service threads 2023-02-09 18:00:30,917 INFO [M:0;jenkins-hbase4:46519] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-09 18:00:30,917 ERROR [M:0;jenkins-hbase4:46519] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-2,5,PEWorkerGroup] 2023-02-09 18:00:30,917 INFO [M:0;jenkins-hbase4:46519] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 18:00:30,917 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-09 18:00:30,918 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-09 18:00:30,918 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 18:00:30,918 DEBUG [M:0;jenkins-hbase4:46519] zookeeper.ZKUtil(398): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-09 18:00:30,918 WARN [M:0;jenkins-hbase4:46519] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-09 18:00:30,918 INFO [M:0;jenkins-hbase4:46519] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 18:00:30,919 INFO [M:0;jenkins-hbase4:46519] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-09 18:00:30,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 18:00:30,919 DEBUG [M:0;jenkins-hbase4:46519] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-09 18:00:30,919 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:30,919 DEBUG [M:0;jenkins-hbase4:46519] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:30,919 DEBUG [M:0;jenkins-hbase4:46519] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-09 18:00:30,919 DEBUG [M:0;jenkins-hbase4:46519] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:30,919 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.07 KB heapSize=29.55 KB 2023-02-09 18:00:30,936 INFO [M:0;jenkins-hbase4:46519] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9c5e0922e466493798f41561dbb3c591 2023-02-09 18:00:30,942 DEBUG [M:0;jenkins-hbase4:46519] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9c5e0922e466493798f41561dbb3c591 as hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9c5e0922e466493798f41561dbb3c591 2023-02-09 18:00:30,948 INFO [M:0;jenkins-hbase4:46519] regionserver.HStore(1080): Added hdfs://localhost:35575/user/jenkins/test-data/4e4a880d-aa26-b58a-ae44-47d100aa4718/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9c5e0922e466493798f41561dbb3c591, entries=8, sequenceid=66, filesize=6.3 K 2023-02-09 18:00:30,949 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegion(2947): Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=66, compaction requested=false 2023-02-09 18:00:30,950 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 18:00:30,950 DEBUG [M:0;jenkins-hbase4:46519] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1675965630919Disabling compacts and flushes for region at 1675965630919Disabling writes for close at 1675965630919Obtaining lock to block concurrent updates at 1675965630919Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1675965630919Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=24646, getHeapSize=30248, getOffHeapSize=0, getCellsCount=71 at 1675965630920 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1675965630920Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1675965630921 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1675965630925 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1675965630925Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1675965630943 (+18 ms)Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=66, compaction requested=false at 1675965630949 (+6 ms)Writing region close event to WAL at 1675965630950 (+1 ms)Closed at 1675965630950 2023-02-09 18:00:30,954 INFO [M:0;jenkins-hbase4:46519] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-09 18:00:30,954 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 18:00:30,955 INFO [M:0;jenkins-hbase4:46519] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46519 2023-02-09 18:00:30,957 DEBUG [M:0;jenkins-hbase4:46519] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46519,1675965628928 already deleted, retry=false 2023-02-09 18:00:30,992 INFO [Listener at localhost/37495] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/37495. 2023-02-09 18:00:30,992 DEBUG [Listener at localhost/37495] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2.tearDown(TestAsyncClusterAdminApi2.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-02-09 18:00:30,992 DEBUG [Listener at localhost/37495] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,993 DEBUG [Listener at localhost/37495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x661a42c5 to 127.0.0.1:50070 2023-02-09 18:00:30,993 INFO [Listener at localhost/37495] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-09 18:00:30,993 DEBUG [Listener at localhost/37495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c7748f1 to 127.0.0.1:50070 2023-02-09 18:00:30,994 DEBUG [Listener at localhost/37495] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 18:00:30,994 DEBUG [Listener at localhost/37495] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-09 18:00:31,093 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:31,093 INFO [M:0;jenkins-hbase4:46519] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46519,1675965628928; zookeeper connection closed. 2023-02-09 18:00:31,095 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@54db2393 rejected from java.util.concurrent.ThreadPoolExecutor@50e4add0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 29] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:31,095 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): master:46519-0x1003a3debac0000, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:31,096 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@29f3c472 rejected from java.util.concurrent.ThreadPoolExecutor@50e4add0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 29] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:31,134 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-09 18:00:31,194 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:31,194 INFO [RS:0;jenkins-hbase4:37375] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37375,1675965628988; zookeeper connection closed. 2023-02-09 18:00:31,194 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14a19df2 rejected from java.util.concurrent.ThreadPoolExecutor@711c0991[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:31,194 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@563df494] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@563df494 2023-02-09 18:00:31,194 DEBUG [Listener at localhost/37495-EventThread] zookeeper.ZKWatcher(600): regionserver:37375-0x1003a3debac0001, quorum=127.0.0.1:50070, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 18:00:31,194 ERROR [Listener at localhost/37495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@10768927 rejected from java.util.concurrent.ThreadPoolExecutor@711c0991[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 18:00:31,194 INFO [Listener at localhost/37495] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-09 18:00:31,195 WARN [Listener at localhost/37495] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 18:00:31,200 INFO [Listener at localhost/37495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:31,304 WARN [BP-603106233-172.31.14.131-1675965628192 heartbeating to localhost/127.0.0.1:35575] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 18:00:31,305 WARN [BP-603106233-172.31.14.131-1675965628192 heartbeating to localhost/127.0.0.1:35575] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-603106233-172.31.14.131-1675965628192 (Datanode Uuid 1820aa52-cc4c-4872-8f61-44f60519f804) service to localhost/127.0.0.1:35575 2023-02-09 18:00:31,305 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/dfs/data/data5/current/BP-603106233-172.31.14.131-1675965628192] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:31,306 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/dfs/data/data6/current/BP-603106233-172.31.14.131-1675965628192] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:31,307 WARN [Listener at localhost/37495] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 18:00:31,310 INFO [Listener at localhost/37495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:31,414 WARN [BP-603106233-172.31.14.131-1675965628192 heartbeating to localhost/127.0.0.1:35575] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 18:00:31,414 WARN [BP-603106233-172.31.14.131-1675965628192 heartbeating to localhost/127.0.0.1:35575] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-603106233-172.31.14.131-1675965628192 (Datanode Uuid e8e292da-c79b-4bc6-943a-d8433902573f) service to localhost/127.0.0.1:35575 2023-02-09 18:00:31,415 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/dfs/data/data3/current/BP-603106233-172.31.14.131-1675965628192] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:31,415 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/dfs/data/data4/current/BP-603106233-172.31.14.131-1675965628192] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:31,416 WARN [Listener at localhost/37495] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 18:00:31,420 INFO [Listener at localhost/37495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:31,523 WARN [BP-603106233-172.31.14.131-1675965628192 heartbeating to localhost/127.0.0.1:35575] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 18:00:31,523 WARN [BP-603106233-172.31.14.131-1675965628192 heartbeating to localhost/127.0.0.1:35575] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-603106233-172.31.14.131-1675965628192 (Datanode Uuid d5ee1a81-9006-40a4-a9c9-30ebebb899e8) service to localhost/127.0.0.1:35575 2023-02-09 18:00:31,524 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/dfs/data/data1/current/BP-603106233-172.31.14.131-1675965628192] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:31,524 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b70bf57d-36a0-1c6b-ef47-1fceead8c8fd/cluster_89b378dc-6ae0-f713-2ebc-6ca2e2d8689e/dfs/data/data2/current/BP-603106233-172.31.14.131-1675965628192] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 18:00:31,537 INFO [Listener at localhost/37495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 18:00:31,651 INFO [Listener at localhost/37495] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-09 18:00:31,675 INFO [Listener at localhost/37495] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-09 18:00:31,687 INFO [Listener at localhost/37495] hbase.ResourceChecker(175): after: client.TestAsyncClusterAdminApi2#testShutdown Thread=111 (was 79) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:35575 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:35575 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:35575 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/37495 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:35575 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:35575 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:35575 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:35575 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:35575 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (985377774) connection to localhost/127.0.0.1:35575 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=529 (was 488) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=324 (was 327), ProcessCount=167 (was 167), AvailableMemoryMB=3180 (was 3263)