2023-02-14 03:53:33,752 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart timeout: 13 mins 2023-02-14 03:53:33,755 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestartSplitWithoutZk timeout: 13 mins 2023-02-14 03:53:33,801 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=7, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=171, AvailableMemoryMB=4238 2023-02-14 03:53:34,069 DEBUG [Time-limited test] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d 2023-02-14 03:53:34,071 INFO [Time-limited test] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-14 03:53:34,075 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-14 03:53:34,075 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38, deleteOnExit=true 2023-02-14 03:53:34,075 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-14 03:53:34,076 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/test.cache.data in system properties and HBase conf 2023-02-14 03:53:34,076 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/hadoop.tmp.dir in system properties and HBase conf 2023-02-14 03:53:34,077 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/hadoop.log.dir in system properties and HBase conf 2023-02-14 03:53:34,077 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-14 03:53:34,078 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-14 03:53:34,078 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-14 03:53:34,180 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-14 03:53:34,577 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-14 03:53:34,581 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-14 03:53:34,581 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-14 03:53:34,581 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-14 03:53:34,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-14 03:53:34,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-14 03:53:34,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-14 03:53:34,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-14 03:53:34,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-14 03:53:34,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-14 03:53:34,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/nfs.dump.dir in system properties and HBase conf 2023-02-14 03:53:34,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/java.io.tmpdir in system properties and HBase conf 2023-02-14 03:53:34,585 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-14 03:53:34,585 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-14 03:53:34,585 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-14 03:53:35,176 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-14 03:53:35,183 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-14 03:53:35,472 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-14 03:53:35,653 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-14 03:53:35,677 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:35,718 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:35,763 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/java.io.tmpdir/Jetty_localhost_37825_hdfs____psqx4x/webapp 2023-02-14 03:53:35,939 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37825 2023-02-14 03:53:35,958 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-14 03:53:35,959 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-14 03:53:36,573 WARN [Listener at localhost/39363] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:36,638 WARN [Listener at localhost/39363] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 03:53:36,656 WARN [Listener at localhost/39363] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:36,662 INFO [Listener at localhost/39363] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:36,667 INFO [Listener at localhost/39363] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/java.io.tmpdir/Jetty_localhost_38457_datanode____i773y0/webapp 2023-02-14 03:53:36,764 INFO [Listener at localhost/39363] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38457 2023-02-14 03:53:37,074 WARN [Listener at localhost/36059] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:37,084 WARN [Listener at localhost/36059] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 03:53:37,087 WARN [Listener at localhost/36059] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:37,089 INFO [Listener at localhost/36059] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:37,093 INFO [Listener at localhost/36059] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/java.io.tmpdir/Jetty_localhost_45091_datanode____wvsxvy/webapp 2023-02-14 03:53:37,192 INFO [Listener at localhost/36059] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45091 2023-02-14 03:53:37,200 WARN [Listener at localhost/38191] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:37,215 WARN [Listener at localhost/38191] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 03:53:37,217 WARN [Listener at localhost/38191] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:37,218 INFO [Listener at localhost/38191] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:37,223 INFO [Listener at localhost/38191] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/java.io.tmpdir/Jetty_localhost_46153_datanode____22wnck/webapp 2023-02-14 03:53:37,324 INFO [Listener at localhost/38191] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46153 2023-02-14 03:53:37,333 WARN [Listener at localhost/44029] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:37,558 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x713af0ce4e43c2c7: Processing first storage report for DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a from datanode 5b6281ff-3c02-403e-8714-a73ce7a86e3f 2023-02-14 03:53:37,560 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x713af0ce4e43c2c7: from storage DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a node DatanodeRegistration(127.0.0.1:44313, datanodeUuid=5b6281ff-3c02-403e-8714-a73ce7a86e3f, infoPort=39027, infoSecurePort=0, ipcPort=36059, storageInfo=lv=-57;cid=testClusterID;nsid=1917091279;c=1676346815259), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-14 03:53:37,560 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf46772583246c4de: Processing first storage report for DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f from datanode 3ce420cf-03fc-4f40-a1ac-24a87b00d91a 2023-02-14 03:53:37,560 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf46772583246c4de: from storage DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f node DatanodeRegistration(127.0.0.1:46195, datanodeUuid=3ce420cf-03fc-4f40-a1ac-24a87b00d91a, infoPort=35109, infoSecurePort=0, ipcPort=38191, storageInfo=lv=-57;cid=testClusterID;nsid=1917091279;c=1676346815259), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc52e147183689822: Processing first storage report for DS-e0682349-ed5b-464f-83b9-71b868c7df59 from datanode e010bf8c-ce27-46b0-b9ee-e8cb58902037 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc52e147183689822: from storage DS-e0682349-ed5b-464f-83b9-71b868c7df59 node DatanodeRegistration(127.0.0.1:39919, datanodeUuid=e010bf8c-ce27-46b0-b9ee-e8cb58902037, infoPort=34903, infoSecurePort=0, ipcPort=44029, storageInfo=lv=-57;cid=testClusterID;nsid=1917091279;c=1676346815259), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x713af0ce4e43c2c7: Processing first storage report for DS-8149f7bf-6b79-4415-bb74-2109d1adc46e from datanode 5b6281ff-3c02-403e-8714-a73ce7a86e3f 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x713af0ce4e43c2c7: from storage DS-8149f7bf-6b79-4415-bb74-2109d1adc46e node DatanodeRegistration(127.0.0.1:44313, datanodeUuid=5b6281ff-3c02-403e-8714-a73ce7a86e3f, infoPort=39027, infoSecurePort=0, ipcPort=36059, storageInfo=lv=-57;cid=testClusterID;nsid=1917091279;c=1676346815259), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf46772583246c4de: Processing first storage report for DS-507dcf92-ecdb-4d6b-b5d4-81b288d1cb1a from datanode 3ce420cf-03fc-4f40-a1ac-24a87b00d91a 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf46772583246c4de: from storage DS-507dcf92-ecdb-4d6b-b5d4-81b288d1cb1a node DatanodeRegistration(127.0.0.1:46195, datanodeUuid=3ce420cf-03fc-4f40-a1ac-24a87b00d91a, infoPort=35109, infoSecurePort=0, ipcPort=38191, storageInfo=lv=-57;cid=testClusterID;nsid=1917091279;c=1676346815259), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-14 03:53:37,561 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc52e147183689822: Processing first storage report for DS-ba4a5468-96af-4fa3-91e3-968e04406c21 from datanode e010bf8c-ce27-46b0-b9ee-e8cb58902037 2023-02-14 03:53:37,562 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc52e147183689822: from storage DS-ba4a5468-96af-4fa3-91e3-968e04406c21 node DatanodeRegistration(127.0.0.1:39919, datanodeUuid=e010bf8c-ce27-46b0-b9ee-e8cb58902037, infoPort=34903, infoSecurePort=0, ipcPort=44029, storageInfo=lv=-57;cid=testClusterID;nsid=1917091279;c=1676346815259), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:37,759 DEBUG [Listener at localhost/44029] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d 2023-02-14 03:53:37,824 INFO [Listener at localhost/44029] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/zookeeper_0, clientPort=60441, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-14 03:53:37,837 INFO [Listener at localhost/44029] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60441 2023-02-14 03:53:37,845 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:37,847 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:38,518 INFO [Listener at localhost/44029] util.FSUtils(479): Created version file at hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83 with version=8 2023-02-14 03:53:38,519 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/hbase-staging 2023-02-14 03:53:38,826 INFO [Listener at localhost/44029] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-14 03:53:39,284 INFO [Listener at localhost/44029] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:39,315 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,315 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,316 INFO [Listener at localhost/44029] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:39,316 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,316 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:39,451 INFO [Listener at localhost/44029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:39,520 DEBUG [Listener at localhost/44029] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-14 03:53:39,609 INFO [Listener at localhost/44029] ipc.NettyRpcServer(120): Bind to /172.31.2.10:35193 2023-02-14 03:53:39,619 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:39,621 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:39,641 INFO [Listener at localhost/44029] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35193 connecting to ZooKeeper ensemble=127.0.0.1:60441 2023-02-14 03:53:39,679 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:351930x0, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:39,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35193-0x10050f75d000000 connected 2023-02-14 03:53:39,706 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:39,707 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:39,710 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:39,717 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35193 2023-02-14 03:53:39,718 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35193 2023-02-14 03:53:39,718 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35193 2023-02-14 03:53:39,719 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35193 2023-02-14 03:53:39,719 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35193 2023-02-14 03:53:39,725 INFO [Listener at localhost/44029] master.HMaster(439): hbase.rootdir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83, hbase.cluster.distributed=false 2023-02-14 03:53:39,793 INFO [Listener at localhost/44029] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:39,793 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,794 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,794 INFO [Listener at localhost/44029] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:39,794 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,794 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:39,799 INFO [Listener at localhost/44029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:39,802 INFO [Listener at localhost/44029] ipc.NettyRpcServer(120): Bind to /172.31.2.10:41249 2023-02-14 03:53:39,805 INFO [Listener at localhost/44029] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:39,812 DEBUG [Listener at localhost/44029] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:39,813 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:39,815 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:39,817 INFO [Listener at localhost/44029] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41249 connecting to ZooKeeper ensemble=127.0.0.1:60441 2023-02-14 03:53:39,934 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:412490x0, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:39,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41249-0x10050f75d000001 connected 2023-02-14 03:53:39,935 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:39,937 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:39,938 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:39,938 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41249 2023-02-14 03:53:39,940 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41249 2023-02-14 03:53:39,941 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41249 2023-02-14 03:53:39,941 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41249 2023-02-14 03:53:39,942 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41249 2023-02-14 03:53:39,954 INFO [Listener at localhost/44029] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:39,954 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,955 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,955 INFO [Listener at localhost/44029] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:39,955 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:39,955 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:39,956 INFO [Listener at localhost/44029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:39,957 INFO [Listener at localhost/44029] ipc.NettyRpcServer(120): Bind to /172.31.2.10:45703 2023-02-14 03:53:39,957 INFO [Listener at localhost/44029] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:39,959 DEBUG [Listener at localhost/44029] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:39,960 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:39,963 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:39,966 INFO [Listener at localhost/44029] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45703 connecting to ZooKeeper ensemble=127.0.0.1:60441 2023-02-14 03:53:40,095 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:457030x0, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:40,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45703-0x10050f75d000002 connected 2023-02-14 03:53:40,097 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:40,098 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:40,099 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:40,100 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45703 2023-02-14 03:53:40,100 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45703 2023-02-14 03:53:40,103 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45703 2023-02-14 03:53:40,107 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45703 2023-02-14 03:53:40,107 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45703 2023-02-14 03:53:40,119 INFO [Listener at localhost/44029] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:40,120 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:40,120 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:40,120 INFO [Listener at localhost/44029] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:40,120 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:40,120 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:40,120 INFO [Listener at localhost/44029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:40,122 INFO [Listener at localhost/44029] ipc.NettyRpcServer(120): Bind to /172.31.2.10:35309 2023-02-14 03:53:40,122 INFO [Listener at localhost/44029] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:40,123 DEBUG [Listener at localhost/44029] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:40,124 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:40,126 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:40,127 INFO [Listener at localhost/44029] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35309 connecting to ZooKeeper ensemble=127.0.0.1:60441 2023-02-14 03:53:40,219 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:353090x0, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:40,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35309-0x10050f75d000003 connected 2023-02-14 03:53:40,220 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:40,221 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:40,222 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:40,223 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35309 2023-02-14 03:53:40,223 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35309 2023-02-14 03:53:40,223 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35309 2023-02-14 03:53:40,224 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35309 2023-02-14 03:53:40,224 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35309 2023-02-14 03:53:40,226 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:40,234 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:40,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:40,253 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:40,253 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:40,253 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:40,253 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:40,253 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:40,254 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:40,256 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,35193,1676346818666 from backup master directory 2023-02-14 03:53:40,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:40,262 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:40,262 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:40,262 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:40,263 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:40,265 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-14 03:53:40,266 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-14 03:53:40,355 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/hbase.id with ID: 91d4bdd4-b4a8-4072-9e0b-d4d994d6ea46 2023-02-14 03:53:40,391 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:40,405 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:40,451 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2450ed04 to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:40,483 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4732e8cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:40,505 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:40,508 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-14 03:53:40,524 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-14 03:53:40,524 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-14 03:53:40,526 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:40,530 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:40,531 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:40,563 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store-tmp 2023-02-14 03:53:40,612 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:40,613 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 03:53:40,613 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:40,613 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:40,613 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 03:53:40,614 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:40,614 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:40,614 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676346820613Disabling compacts and flushes for region at 1676346820613Disabling writes for close at 1676346820614 (+1 ms)Writing region close event to WAL at 1676346820614Closed at 1676346820614 2023-02-14 03:53:40,616 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:40,635 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C35193%2C1676346818666, suffix=, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/oldWALs, maxLogs=10 2023-02-14 03:53:40,689 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:40,689 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:40,689 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:40,697 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:40,757 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 2023-02-14 03:53:40,757 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK], DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK], DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK]] 2023-02-14 03:53:40,758 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:40,758 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:40,761 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:40,762 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:40,819 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:40,825 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-14 03:53:40,848 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-14 03:53:40,862 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:40,867 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:40,869 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:40,882 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:40,886 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:40,887 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59515737, jitterRate=-0.11314640939235687}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:40,887 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676346820787Initializing all the Stores at 1676346820789 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346820789Cleaning up temporary data from old regions at 1676346820874 (+85 ms)Cleaning up detritus from prior splits at 1676346820875 (+1 ms)Region opened successfully at 1676346820887 (+12 ms) 2023-02-14 03:53:40,888 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-14 03:53:40,906 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-14 03:53:40,906 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-14 03:53:40,908 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-14 03:53:40,909 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-14 03:53:40,940 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 30 msec 2023-02-14 03:53:40,940 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-14 03:53:40,964 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-14 03:53:40,970 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-14 03:53:40,992 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-14 03:53:40,996 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-14 03:53:40,997 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-14 03:53:41,002 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-14 03:53:41,005 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-14 03:53:41,008 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:41,009 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-14 03:53:41,010 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-14 03:53:41,023 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-14 03:53:41,029 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:41,029 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:41,029 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:41,029 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:41,029 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:41,029 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase9.apache.org,35193,1676346818666, sessionid=0x10050f75d000000, setting cluster-up flag (Was=false) 2023-02-14 03:53:41,047 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:41,053 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-14 03:53:41,055 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:41,059 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:41,064 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-14 03:53:41,066 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:41,068 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.hbase-snapshot/.tmp 2023-02-14 03:53:41,128 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(952): ClusterId : 91d4bdd4-b4a8-4072-9e0b-d4d994d6ea46 2023-02-14 03:53:41,128 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(952): ClusterId : 91d4bdd4-b4a8-4072-9e0b-d4d994d6ea46 2023-02-14 03:53:41,128 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(952): ClusterId : 91d4bdd4-b4a8-4072-9e0b-d4d994d6ea46 2023-02-14 03:53:41,135 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:41,135 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:41,135 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:41,141 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:41,141 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:41,141 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:41,142 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:41,142 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:41,142 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:41,145 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:41,145 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:41,147 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:41,148 DEBUG [RS:2;jenkins-hbase9:35309] zookeeper.ReadOnlyZKClient(139): Connect 0x33bf36bb to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:41,148 DEBUG [RS:0;jenkins-hbase9:41249] zookeeper.ReadOnlyZKClient(139): Connect 0x42c21170 to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:41,148 DEBUG [RS:1;jenkins-hbase9:45703] zookeeper.ReadOnlyZKClient(139): Connect 0x294828c6 to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:41,156 DEBUG [RS:2;jenkins-hbase9:35309] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d78cebc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:41,157 DEBUG [RS:2;jenkins-hbase9:35309] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39475dcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:41,158 DEBUG [RS:1;jenkins-hbase9:45703] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@278fc39f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:41,158 DEBUG [RS:0;jenkins-hbase9:41249] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c05f758, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:41,158 DEBUG [RS:1;jenkins-hbase9:45703] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23b043b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:41,158 DEBUG [RS:0;jenkins-hbase9:41249] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@483a7a00, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:41,185 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase9:45703 2023-02-14 03:53:41,186 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase9:35309 2023-02-14 03:53:41,188 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:41249 2023-02-14 03:53:41,188 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:41,189 INFO [RS:1;jenkins-hbase9:45703] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:41,189 INFO [RS:1;jenkins-hbase9:45703] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:41,189 INFO [RS:2;jenkins-hbase9:35309] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:41,190 INFO [RS:0;jenkins-hbase9:41249] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:41,190 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:41,190 INFO [RS:0;jenkins-hbase9:41249] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:41,190 INFO [RS:2;jenkins-hbase9:35309] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:41,190 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:41,190 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:41,193 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,35193,1676346818666 with isa=jenkins-hbase9.apache.org/172.31.2.10:35309, startcode=1676346820119 2023-02-14 03:53:41,193 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,35193,1676346818666 with isa=jenkins-hbase9.apache.org/172.31.2.10:45703, startcode=1676346819954 2023-02-14 03:53:41,193 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,35193,1676346818666 with isa=jenkins-hbase9.apache.org/172.31.2.10:41249, startcode=1676346819792 2023-02-14 03:53:41,200 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:41,200 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:41,201 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:41,201 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:41,201 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-02-14 03:53:41,201 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,201 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:41,201 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,203 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676346851203 2023-02-14 03:53:41,205 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-14 03:53:41,210 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:41,210 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-14 03:53:41,213 DEBUG [RS:0;jenkins-hbase9:41249] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:41,213 DEBUG [RS:1;jenkins-hbase9:45703] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:41,213 DEBUG [RS:2;jenkins-hbase9:35309] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:41,216 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-14 03:53:41,216 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:41,223 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-14 03:53:41,223 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-14 03:53:41,224 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-14 03:53:41,224 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-14 03:53:41,224 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,226 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-14 03:53:41,227 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-14 03:53:41,228 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-14 03:53:41,231 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-14 03:53:41,231 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-14 03:53:41,233 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346821233,5,FailOnTimeoutGroup] 2023-02-14 03:53:41,239 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346821233,5,FailOnTimeoutGroup] 2023-02-14 03:53:41,239 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,241 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-14 03:53:41,244 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,244 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,267 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:41,269 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:41,269 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83 2023-02-14 03:53:41,282 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36851, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:41,282 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36893, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:41,282 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34325, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:41,294 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:41,297 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 03:53:41,300 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/info 2023-02-14 03:53:41,301 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35193] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,302 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35193] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,302 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 03:53:41,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:41,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 03:53:41,305 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:41,306 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 03:53:41,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:41,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 03:53:41,309 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/table 2023-02-14 03:53:41,310 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 03:53:41,310 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:41,312 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740 2023-02-14 03:53:41,313 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740 2023-02-14 03:53:41,316 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 03:53:41,318 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83 2023-02-14 03:53:41,318 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83 2023-02-14 03:53:41,318 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83 2023-02-14 03:53:41,318 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39363 2023-02-14 03:53:41,318 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39363 2023-02-14 03:53:41,318 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:41,318 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39363 2023-02-14 03:53:41,318 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:41,319 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:41,320 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 03:53:41,324 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:41,325 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62429140, jitterRate=-0.06973332166671753}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 03:53:41,325 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676346821295Initializing all the Stores at 1676346821296 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346821297 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346821297Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346821297Cleaning up temporary data from old regions at 1676346821314 (+17 ms)Cleaning up detritus from prior splits at 1676346821315 (+1 ms)Region opened successfully at 1676346821325 (+10 ms) 2023-02-14 03:53:41,325 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 03:53:41,325 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 03:53:41,325 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 03:53:41,325 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 03:53:41,325 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 03:53:41,327 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 03:53:41,327 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676346821325Disabling compacts and flushes for region at 1676346821325Disabling writes for close at 1676346821325Writing region close event to WAL at 1676346821327 (+2 ms)Closed at 1676346821327 2023-02-14 03:53:41,328 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:41,329 DEBUG [RS:2;jenkins-hbase9:35309] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,329 DEBUG [RS:1;jenkins-hbase9:45703] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,329 DEBUG [RS:0;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,329 WARN [RS:1;jenkins-hbase9:45703] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:41,330 INFO [RS:1;jenkins-hbase9:45703] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:41,330 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,331 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,41249,1676346819792] 2023-02-14 03:53:41,329 WARN [RS:2;jenkins-hbase9:35309] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:41,331 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,35309,1676346820119] 2023-02-14 03:53:41,330 WARN [RS:0;jenkins-hbase9:41249] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:41,331 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,45703,1676346819954] 2023-02-14 03:53:41,334 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:41,334 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-14 03:53:41,331 INFO [RS:2;jenkins-hbase9:35309] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:41,331 INFO [RS:0;jenkins-hbase9:41249] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:41,336 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,336 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-14 03:53:41,358 DEBUG [RS:0;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,358 DEBUG [RS:1;jenkins-hbase9:45703] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,359 DEBUG [RS:2;jenkins-hbase9:35309] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,359 DEBUG [RS:0;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,359 DEBUG [RS:1;jenkins-hbase9:45703] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,359 DEBUG [RS:2;jenkins-hbase9:35309] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,360 DEBUG [RS:0;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,360 DEBUG [RS:1;jenkins-hbase9:45703] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,360 DEBUG [RS:2;jenkins-hbase9:35309] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,371 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-14 03:53:41,371 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:41,371 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:41,371 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:41,374 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-14 03:53:41,382 INFO [RS:1;jenkins-hbase9:45703] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:41,382 INFO [RS:0;jenkins-hbase9:41249] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:41,383 INFO [RS:2;jenkins-hbase9:35309] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:41,401 INFO [RS:2;jenkins-hbase9:35309] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:41,401 INFO [RS:0;jenkins-hbase9:41249] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:41,401 INFO [RS:1;jenkins-hbase9:45703] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:41,408 INFO [RS:2;jenkins-hbase9:35309] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:41,408 INFO [RS:0;jenkins-hbase9:41249] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:41,409 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,409 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,408 INFO [RS:1;jenkins-hbase9:45703] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:41,409 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,410 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:41,410 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:41,410 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:41,418 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,418 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,418 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,418 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,418 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:41,419 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,419 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:2;jenkins-hbase9:35309] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:0;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,420 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,421 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:41,421 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,421 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,421 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,422 DEBUG [RS:1;jenkins-hbase9:45703] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:41,423 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,424 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,424 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,424 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,424 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,424 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,424 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,425 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,425 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,441 INFO [RS:2;jenkins-hbase9:35309] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:41,441 INFO [RS:0;jenkins-hbase9:41249] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:41,441 INFO [RS:1;jenkins-hbase9:45703] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:41,444 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35309,1676346820119-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,444 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45703,1676346819954-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,444 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41249,1676346819792-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,466 INFO [RS:1;jenkins-hbase9:45703] regionserver.Replication(203): jenkins-hbase9.apache.org,45703,1676346819954 started 2023-02-14 03:53:41,466 INFO [RS:0;jenkins-hbase9:41249] regionserver.Replication(203): jenkins-hbase9.apache.org,41249,1676346819792 started 2023-02-14 03:53:41,466 INFO [RS:2;jenkins-hbase9:35309] regionserver.Replication(203): jenkins-hbase9.apache.org,35309,1676346820119 started 2023-02-14 03:53:41,466 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,45703,1676346819954, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:45703, sessionid=0x10050f75d000002 2023-02-14 03:53:41,467 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,35309,1676346820119, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:35309, sessionid=0x10050f75d000003 2023-02-14 03:53:41,467 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,41249,1676346819792, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:41249, sessionid=0x10050f75d000001 2023-02-14 03:53:41,467 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:41,467 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:41,467 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:41,467 DEBUG [RS:1;jenkins-hbase9:45703] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,467 DEBUG [RS:0;jenkins-hbase9:41249] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,468 DEBUG [RS:1;jenkins-hbase9:45703] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,45703,1676346819954' 2023-02-14 03:53:41,468 DEBUG [RS:2;jenkins-hbase9:35309] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,468 DEBUG [RS:1;jenkins-hbase9:45703] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:41,468 DEBUG [RS:0;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,41249,1676346819792' 2023-02-14 03:53:41,468 DEBUG [RS:2;jenkins-hbase9:35309] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,35309,1676346820119' 2023-02-14 03:53:41,468 DEBUG [RS:0;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:41,468 DEBUG [RS:2;jenkins-hbase9:35309] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:41,469 DEBUG [RS:1;jenkins-hbase9:45703] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:41,469 DEBUG [RS:0;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:41,469 DEBUG [RS:2;jenkins-hbase9:35309] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:41,469 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:41,469 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:41,469 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:41,470 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:41,470 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:41,470 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:41,470 DEBUG [RS:2;jenkins-hbase9:35309] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:41,470 DEBUG [RS:1;jenkins-hbase9:45703] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,470 DEBUG [RS:2;jenkins-hbase9:35309] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,35309,1676346820119' 2023-02-14 03:53:41,470 DEBUG [RS:0;jenkins-hbase9:41249] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:41,471 DEBUG [RS:0;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,41249,1676346819792' 2023-02-14 03:53:41,471 DEBUG [RS:0;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:41,470 DEBUG [RS:2;jenkins-hbase9:35309] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:41,470 DEBUG [RS:1;jenkins-hbase9:45703] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,45703,1676346819954' 2023-02-14 03:53:41,471 DEBUG [RS:1;jenkins-hbase9:45703] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:41,471 DEBUG [RS:0;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:41,471 DEBUG [RS:2;jenkins-hbase9:35309] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:41,472 DEBUG [RS:0;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:41,472 DEBUG [RS:1;jenkins-hbase9:45703] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:41,472 INFO [RS:0;jenkins-hbase9:41249] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:41,472 DEBUG [RS:2;jenkins-hbase9:35309] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:41,472 INFO [RS:0;jenkins-hbase9:41249] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:41,472 INFO [RS:2;jenkins-hbase9:35309] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:41,472 DEBUG [RS:1;jenkins-hbase9:45703] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:41,473 INFO [RS:2;jenkins-hbase9:35309] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:41,474 INFO [RS:1;jenkins-hbase9:45703] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:41,474 INFO [RS:1;jenkins-hbase9:45703] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:41,526 DEBUG [jenkins-hbase9:35193] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-14 03:53:41,529 DEBUG [jenkins-hbase9:35193] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:41,535 DEBUG [jenkins-hbase9:35193] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:41,535 DEBUG [jenkins-hbase9:35193] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:41,535 DEBUG [jenkins-hbase9:35193] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:41,535 DEBUG [jenkins-hbase9:35193] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:41,538 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,45703,1676346819954, state=OPENING 2023-02-14 03:53:41,546 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-14 03:53:41,547 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:41,548 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 03:53:41,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,45703,1676346819954}] 2023-02-14 03:53:41,583 INFO [RS:0;jenkins-hbase9:41249] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C41249%2C1676346819792, suffix=, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs, maxLogs=32 2023-02-14 03:53:41,583 INFO [RS:2;jenkins-hbase9:35309] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C35309%2C1676346820119, suffix=, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,35309,1676346820119, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs, maxLogs=32 2023-02-14 03:53:41,583 INFO [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C45703%2C1676346819954, suffix=, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,45703,1676346819954, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs, maxLogs=32 2023-02-14 03:53:41,605 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:41,605 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:41,605 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:41,611 INFO [RS:0;jenkins-hbase9:41249] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792/jenkins-hbase9.apache.org%2C41249%2C1676346819792.1676346821587 2023-02-14 03:53:41,615 DEBUG [RS:0;jenkins-hbase9:41249] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK], DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK], DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK]] 2023-02-14 03:53:41,617 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:41,617 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:41,618 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:41,624 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:41,624 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:41,631 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:41,640 INFO [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,45703,1676346819954/jenkins-hbase9.apache.org%2C45703%2C1676346819954.1676346821588 2023-02-14 03:53:41,642 DEBUG [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK], DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK], DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK]] 2023-02-14 03:53:41,642 INFO [RS:2;jenkins-hbase9:35309] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,35309,1676346820119/jenkins-hbase9.apache.org%2C35309%2C1676346820119.1676346821588 2023-02-14 03:53:41,642 DEBUG [RS:2;jenkins-hbase9:35309] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK], DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK], DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK]] 2023-02-14 03:53:41,735 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:41,738 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:41,741 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 03:53:41,753 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-14 03:53:41,754 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:41,759 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C45703%2C1676346819954.meta, suffix=.meta, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,45703,1676346819954, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs, maxLogs=32 2023-02-14 03:53:41,778 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:41,778 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:41,780 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:41,785 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,45703,1676346819954/jenkins-hbase9.apache.org%2C45703%2C1676346819954.meta.1676346821760.meta 2023-02-14 03:53:41,786 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK], DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK], DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK]] 2023-02-14 03:53:41,786 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:41,788 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-14 03:53:41,806 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-14 03:53:41,812 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-14 03:53:41,817 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-14 03:53:41,817 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:41,817 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-14 03:53:41,817 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-14 03:53:41,820 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 03:53:41,822 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/info 2023-02-14 03:53:41,822 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/info 2023-02-14 03:53:41,822 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 03:53:41,823 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:41,823 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 03:53:41,824 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:41,825 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:41,825 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 03:53:41,826 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:41,826 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 03:53:41,827 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/table 2023-02-14 03:53:41,827 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/table 2023-02-14 03:53:41,828 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 03:53:41,828 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:41,831 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740 2023-02-14 03:53:41,833 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740 2023-02-14 03:53:41,836 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 03:53:41,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 03:53:41,839 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68991730, jitterRate=0.02805688977241516}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 03:53:41,840 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676346821817Writing region info on filesystem at 1676346821818 (+1 ms)Initializing all the Stores at 1676346821819 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346821819Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346821820 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346821820Cleaning up temporary data from old regions at 1676346821834 (+14 ms)Cleaning up detritus from prior splits at 1676346821835 (+1 ms)Running coprocessor post-open hooks at 1676346821840 (+5 ms)Region opened successfully at 1676346821840 2023-02-14 03:53:41,850 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676346821728 2023-02-14 03:53:41,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-14 03:53:41,870 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-14 03:53:41,870 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,45703,1676346819954, state=OPEN 2023-02-14 03:53:41,873 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-14 03:53:41,873 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 03:53:41,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-14 03:53:41,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,45703,1676346819954 in 322 msec 2023-02-14 03:53:41,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-14 03:53:41,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 521 msec 2023-02-14 03:53:41,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 777 msec 2023-02-14 03:53:41,888 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676346820249, completionTime=-1 2023-02-14 03:53:41,888 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-14 03:53:41,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-14 03:53:41,947 DEBUG [hconnection-0x222388e5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:41,950 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:49000, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:41,967 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-14 03:53:41,967 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676346881967 2023-02-14 03:53:41,968 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676346941968 2023-02-14 03:53:41,968 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 79 msec 2023-02-14 03:53:41,994 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35193,1676346818666-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,994 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35193,1676346818666-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,994 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35193,1676346818666-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,996 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:35193, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:41,996 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:42,005 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-14 03:53:42,013 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-14 03:53:42,014 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:42,023 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-14 03:53:42,025 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:42,028 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:42,048 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,050 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2 empty. 2023-02-14 03:53:42,051 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,051 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-14 03:53:42,089 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:42,091 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 899a4f097a47243a194517a0f9f5faf2, NAME => 'hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp 2023-02-14 03:53:42,109 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:42,109 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 899a4f097a47243a194517a0f9f5faf2, disabling compactions & flushes 2023-02-14 03:53:42,109 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,109 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,109 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. after waiting 0 ms 2023-02-14 03:53:42,109 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,109 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,109 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 899a4f097a47243a194517a0f9f5faf2: Waiting for close lock at 1676346822109Disabling compacts and flushes for region at 1676346822109Disabling writes for close at 1676346822109Writing region close event to WAL at 1676346822109Closed at 1676346822109 2023-02-14 03:53:42,114 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:42,128 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346822116"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346822116"}]},"ts":"1676346822116"} 2023-02-14 03:53:42,153 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:42,155 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:42,159 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346822155"}]},"ts":"1676346822155"} 2023-02-14 03:53:42,164 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-14 03:53:42,168 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:42,170 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:42,170 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:42,170 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:42,170 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:42,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=899a4f097a47243a194517a0f9f5faf2, ASSIGN}] 2023-02-14 03:53:42,176 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=899a4f097a47243a194517a0f9f5faf2, ASSIGN 2023-02-14 03:53:42,178 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=899a4f097a47243a194517a0f9f5faf2, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35309,1676346820119; forceNewPlan=false, retain=false 2023-02-14 03:53:42,330 INFO [jenkins-hbase9:35193] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:42,331 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=899a4f097a47243a194517a0f9f5faf2, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:42,331 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346822330"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346822330"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346822330"}]},"ts":"1676346822330"} 2023-02-14 03:53:42,336 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 899a4f097a47243a194517a0f9f5faf2, server=jenkins-hbase9.apache.org,35309,1676346820119}] 2023-02-14 03:53:42,490 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:42,490 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:42,494 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:57950, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 03:53:42,502 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,504 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 899a4f097a47243a194517a0f9f5faf2, NAME => 'hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:42,504 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,505 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:42,505 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,505 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,506 INFO [StoreOpener-899a4f097a47243a194517a0f9f5faf2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,509 DEBUG [StoreOpener-899a4f097a47243a194517a0f9f5faf2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/info 2023-02-14 03:53:42,509 DEBUG [StoreOpener-899a4f097a47243a194517a0f9f5faf2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/info 2023-02-14 03:53:42,509 INFO [StoreOpener-899a4f097a47243a194517a0f9f5faf2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899a4f097a47243a194517a0f9f5faf2 columnFamilyName info 2023-02-14 03:53:42,510 INFO [StoreOpener-899a4f097a47243a194517a0f9f5faf2-1] regionserver.HStore(310): Store=899a4f097a47243a194517a0f9f5faf2/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:42,512 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,513 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,517 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:42,521 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:42,522 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 899a4f097a47243a194517a0f9f5faf2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67573008, jitterRate=0.0069162845611572266}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:42,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 899a4f097a47243a194517a0f9f5faf2: Running coprocessor pre-open hook at 1676346822505Writing region info on filesystem at 1676346822505Initializing all the Stores at 1676346822506 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346822506Cleaning up temporary data from old regions at 1676346822513 (+7 ms)Cleaning up detritus from prior splits at 1676346822514 (+1 ms)Running coprocessor post-open hooks at 1676346822522 (+8 ms)Region opened successfully at 1676346822522 2023-02-14 03:53:42,524 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2., pid=6, masterSystemTime=1676346822490 2023-02-14 03:53:42,528 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,529 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:42,530 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=899a4f097a47243a194517a0f9f5faf2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:42,531 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346822530"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346822530"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346822530"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346822530"}]},"ts":"1676346822530"} 2023-02-14 03:53:42,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-14 03:53:42,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 899a4f097a47243a194517a0f9f5faf2, server=jenkins-hbase9.apache.org,35309,1676346820119 in 198 msec 2023-02-14 03:53:42,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-14 03:53:42,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=899a4f097a47243a194517a0f9f5faf2, ASSIGN in 364 msec 2023-02-14 03:53:42,543 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:42,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346822543"}]},"ts":"1676346822543"} 2023-02-14 03:53:42,545 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-14 03:53:42,548 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:42,551 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 533 msec 2023-02-14 03:53:42,626 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-14 03:53:42,629 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-14 03:53:42,629 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:42,649 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:42,651 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:57958, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:42,663 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-14 03:53:42,680 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-14 03:53:42,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-02-14 03:53:42,697 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-14 03:53:42,708 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-14 03:53:42,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-14 03:53:42,722 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-14 03:53:42,726 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-14 03:53:42,726 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.463sec 2023-02-14 03:53:42,729 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-14 03:53:42,730 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-14 03:53:42,730 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-14 03:53:42,732 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35193,1676346818666-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-14 03:53:42,734 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35193,1676346818666-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-14 03:53:42,736 DEBUG [Listener at localhost/44029] zookeeper.ReadOnlyZKClient(139): Connect 0x106f9e57 to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:42,741 DEBUG [Listener at localhost/44029] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cb7242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:42,743 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-14 03:53:42,757 DEBUG [hconnection-0x3535bc5e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:42,769 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:49016, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:42,780 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:42,787 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-14 03:53:42,787 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:42,788 INFO [Listener at localhost/44029] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-14 03:53:42,788 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-14 03:53:42,798 DEBUG [Listener at localhost/44029] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-14 03:53:42,802 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-14 03:53:42,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.HMaster$4(2102): Client=jenkins//172.31.2.10 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:42,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-14 03:53:42,818 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:42,820 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:42,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-14 03:53:42,823 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:42,824 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 empty. 2023-02-14 03:53:42,826 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:42,826 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-14 03:53:42,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:42,849 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:42,850 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => 8fe0caa604acbdf24b118bc36deabcf4, NAME => 'restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp 2023-02-14 03:53:42,868 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:42,868 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing 8fe0caa604acbdf24b118bc36deabcf4, disabling compactions & flushes 2023-02-14 03:53:42,868 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:42,868 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:42,868 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. after waiting 0 ms 2023-02-14 03:53:42,868 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:42,868 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:42,868 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for 8fe0caa604acbdf24b118bc36deabcf4: Waiting for close lock at 1676346822868Disabling compacts and flushes for region at 1676346822868Disabling writes for close at 1676346822868Writing region close event to WAL at 1676346822868Closed at 1676346822868 2023-02-14 03:53:42,872 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:42,874 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346822874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346822874"}]},"ts":"1676346822874"} 2023-02-14 03:53:42,877 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:42,878 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:42,878 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346822878"}]},"ts":"1676346822878"} 2023-02-14 03:53:42,880 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-14 03:53:42,885 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:42,886 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:42,886 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:42,886 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:42,886 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:42,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN}] 2023-02-14 03:53:42,889 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN 2023-02-14 03:53:42,890 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,41249,1676346819792; forceNewPlan=false, retain=false 2023-02-14 03:53:42,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:43,041 INFO [jenkins-hbase9:35193] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:43,042 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=8fe0caa604acbdf24b118bc36deabcf4, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:43,042 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346823042"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346823042"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346823042"}]},"ts":"1676346823042"} 2023-02-14 03:53:43,045 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 8fe0caa604acbdf24b118bc36deabcf4, server=jenkins-hbase9.apache.org,41249,1676346819792}] 2023-02-14 03:53:43,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:43,199 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:43,199 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:43,201 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33186, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 03:53:43,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:43,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8fe0caa604acbdf24b118bc36deabcf4, NAME => 'restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:43,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:43,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,209 INFO [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,211 DEBUG [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/family 2023-02-14 03:53:43,212 DEBUG [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/family 2023-02-14 03:53:43,212 INFO [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fe0caa604acbdf24b118bc36deabcf4 columnFamilyName family 2023-02-14 03:53:43,213 INFO [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] regionserver.HStore(310): Store=8fe0caa604acbdf24b118bc36deabcf4/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:43,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:43,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:43,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 8fe0caa604acbdf24b118bc36deabcf4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68611422, jitterRate=0.022389858961105347}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:43,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 8fe0caa604acbdf24b118bc36deabcf4: Running coprocessor pre-open hook at 1676346823208Writing region info on filesystem at 1676346823208Initializing all the Stores at 1676346823209 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346823209Cleaning up temporary data from old regions at 1676346823215 (+6 ms)Cleaning up detritus from prior splits at 1676346823216 (+1 ms)Running coprocessor post-open hooks at 1676346823223 (+7 ms)Region opened successfully at 1676346823223 2023-02-14 03:53:43,224 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4., pid=11, masterSystemTime=1676346823199 2023-02-14 03:53:43,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:43,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:43,229 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=8fe0caa604acbdf24b118bc36deabcf4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:43,229 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346823229"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346823229"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346823229"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346823229"}]},"ts":"1676346823229"} 2023-02-14 03:53:43,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-14 03:53:43,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 8fe0caa604acbdf24b118bc36deabcf4, server=jenkins-hbase9.apache.org,41249,1676346819792 in 187 msec 2023-02-14 03:53:43,239 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-14 03:53:43,239 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN in 349 msec 2023-02-14 03:53:43,240 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:43,241 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346823240"}]},"ts":"1676346823240"} 2023-02-14 03:53:43,243 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-14 03:53:43,246 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:43,248 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 432 msec 2023-02-14 03:53:43,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:43,442 INFO [Listener at localhost/44029] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-14 03:53:43,442 DEBUG [Listener at localhost/44029] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-14 03:53:43,444 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:43,448 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-14 03:53:43,449 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:43,449 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-14 03:53:43,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.HMaster$4(2102): Client=jenkins//172.31.2.10 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-14 03:53:43,454 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:43,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-14 03:53:43,456 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:43,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:43,458 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,459 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 empty. 2023-02-14 03:53:43,461 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,461 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-14 03:53:43,484 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:43,485 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 6babb91185b87642e80cebc762b9f388, NAME => 'restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp 2023-02-14 03:53:43,499 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:43,499 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 6babb91185b87642e80cebc762b9f388, disabling compactions & flushes 2023-02-14 03:53:43,499 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,499 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,499 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. after waiting 0 ms 2023-02-14 03:53:43,499 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,499 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,499 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 6babb91185b87642e80cebc762b9f388: Waiting for close lock at 1676346823499Disabling compacts and flushes for region at 1676346823499Disabling writes for close at 1676346823499Writing region close event to WAL at 1676346823499Closed at 1676346823499 2023-02-14 03:53:43,503 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:43,505 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346823504"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346823504"}]},"ts":"1676346823504"} 2023-02-14 03:53:43,507 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:43,508 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:43,509 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346823508"}]},"ts":"1676346823508"} 2023-02-14 03:53:43,510 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-14 03:53:43,515 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:43,516 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:43,516 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:43,516 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:43,516 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:43,516 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN}] 2023-02-14 03:53:43,518 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN 2023-02-14 03:53:43,519 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,41249,1676346819792; forceNewPlan=false, retain=false 2023-02-14 03:53:43,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:43,669 INFO [jenkins-hbase9:35193] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:43,670 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6babb91185b87642e80cebc762b9f388, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:43,671 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346823670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346823670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346823670"}]},"ts":"1676346823670"} 2023-02-14 03:53:43,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 6babb91185b87642e80cebc762b9f388, server=jenkins-hbase9.apache.org,41249,1676346819792}] 2023-02-14 03:53:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:43,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6babb91185b87642e80cebc762b9f388, NAME => 'restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,833 INFO [StoreOpener-6babb91185b87642e80cebc762b9f388-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,835 DEBUG [StoreOpener-6babb91185b87642e80cebc762b9f388-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/family 2023-02-14 03:53:43,835 DEBUG [StoreOpener-6babb91185b87642e80cebc762b9f388-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/family 2023-02-14 03:53:43,836 INFO [StoreOpener-6babb91185b87642e80cebc762b9f388-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6babb91185b87642e80cebc762b9f388 columnFamilyName family 2023-02-14 03:53:43,836 INFO [StoreOpener-6babb91185b87642e80cebc762b9f388-1] regionserver.HStore(310): Store=6babb91185b87642e80cebc762b9f388/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:43,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:43,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:43,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 6babb91185b87642e80cebc762b9f388; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64372325, jitterRate=-0.04077760875225067}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:43,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 6babb91185b87642e80cebc762b9f388: Running coprocessor pre-open hook at 1676346823832Writing region info on filesystem at 1676346823832Initializing all the Stores at 1676346823833 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346823833Cleaning up temporary data from old regions at 1676346823842 (+9 ms)Cleaning up detritus from prior splits at 1676346823842Running coprocessor post-open hooks at 1676346823848 (+6 ms)Region opened successfully at 1676346823848 2023-02-14 03:53:43,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388., pid=14, masterSystemTime=1676346823826 2023-02-14 03:53:43,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:43,854 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6babb91185b87642e80cebc762b9f388, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:43,854 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346823853"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346823853"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346823853"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346823853"}]},"ts":"1676346823853"} 2023-02-14 03:53:43,860 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-14 03:53:43,860 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 6babb91185b87642e80cebc762b9f388, server=jenkins-hbase9.apache.org,41249,1676346819792 in 184 msec 2023-02-14 03:53:43,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-14 03:53:43,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN in 344 msec 2023-02-14 03:53:43,864 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:43,865 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346823865"}]},"ts":"1676346823865"} 2023-02-14 03:53:43,867 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-14 03:53:43,869 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:43,871 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 418 msec 2023-02-14 03:53:44,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:44,061 INFO [Listener at localhost/44029] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-14 03:53:44,061 DEBUG [Listener at localhost/44029] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-14 03:53:44,062 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,065 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-14 03:53:44,065 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,066 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-14 03:53:44,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.HMaster$4(2102): Client=jenkins//172.31.2.10 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:44,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-14 03:53:44,071 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:44,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-14 03:53:44,072 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:44,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:44,074 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,075 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491 empty. 2023-02-14 03:53:44,076 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,076 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-14 03:53:44,094 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:44,095 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => e4ece1b6b0b8b7a536a6b6b88716f491, NAME => 'restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.tmp 2023-02-14 03:53:44,109 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:44,110 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing e4ece1b6b0b8b7a536a6b6b88716f491, disabling compactions & flushes 2023-02-14 03:53:44,110 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,110 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,110 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. after waiting 0 ms 2023-02-14 03:53:44,110 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,110 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,110 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for e4ece1b6b0b8b7a536a6b6b88716f491: Waiting for close lock at 1676346824110Disabling compacts and flushes for region at 1676346824110Disabling writes for close at 1676346824110Writing region close event to WAL at 1676346824110Closed at 1676346824110 2023-02-14 03:53:44,113 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:44,115 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346824115"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346824115"}]},"ts":"1676346824115"} 2023-02-14 03:53:44,117 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:44,119 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:44,119 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346824119"}]},"ts":"1676346824119"} 2023-02-14 03:53:44,121 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-14 03:53:44,124 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:44,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:44,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:44,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:44,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:44,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=e4ece1b6b0b8b7a536a6b6b88716f491, ASSIGN}] 2023-02-14 03:53:44,127 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=e4ece1b6b0b8b7a536a6b6b88716f491, ASSIGN 2023-02-14 03:53:44,129 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=e4ece1b6b0b8b7a536a6b6b88716f491, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35309,1676346820119; forceNewPlan=false, retain=false 2023-02-14 03:53:44,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:44,279 INFO [jenkins-hbase9:35193] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:44,280 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=e4ece1b6b0b8b7a536a6b6b88716f491, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:44,280 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346824280"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346824280"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346824280"}]},"ts":"1676346824280"} 2023-02-14 03:53:44,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure e4ece1b6b0b8b7a536a6b6b88716f491, server=jenkins-hbase9.apache.org,35309,1676346820119}] 2023-02-14 03:53:44,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:44,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e4ece1b6b0b8b7a536a6b6b88716f491, NAME => 'restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:44,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:44,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,443 INFO [StoreOpener-e4ece1b6b0b8b7a536a6b6b88716f491-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,445 DEBUG [StoreOpener-e4ece1b6b0b8b7a536a6b6b88716f491-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491/family 2023-02-14 03:53:44,445 DEBUG [StoreOpener-e4ece1b6b0b8b7a536a6b6b88716f491-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491/family 2023-02-14 03:53:44,446 INFO [StoreOpener-e4ece1b6b0b8b7a536a6b6b88716f491-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4ece1b6b0b8b7a536a6b6b88716f491 columnFamilyName family 2023-02-14 03:53:44,447 INFO [StoreOpener-e4ece1b6b0b8b7a536a6b6b88716f491-1] regionserver.HStore(310): Store=e4ece1b6b0b8b7a536a6b6b88716f491/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:44,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:44,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:44,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened e4ece1b6b0b8b7a536a6b6b88716f491; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67002507, jitterRate=-0.0015848428010940552}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:44,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for e4ece1b6b0b8b7a536a6b6b88716f491: Running coprocessor pre-open hook at 1676346824442Writing region info on filesystem at 1676346824442Initializing all the Stores at 1676346824443 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346824443Cleaning up temporary data from old regions at 1676346824450 (+7 ms)Cleaning up detritus from prior splits at 1676346824450Running coprocessor post-open hooks at 1676346824459 (+9 ms)Region opened successfully at 1676346824459 2023-02-14 03:53:44,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491., pid=17, masterSystemTime=1676346824436 2023-02-14 03:53:44,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:44,463 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=e4ece1b6b0b8b7a536a6b6b88716f491, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:44,464 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346824463"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346824463"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346824463"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346824463"}]},"ts":"1676346824463"} 2023-02-14 03:53:44,469 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-14 03:53:44,469 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure e4ece1b6b0b8b7a536a6b6b88716f491, server=jenkins-hbase9.apache.org,35309,1676346820119 in 183 msec 2023-02-14 03:53:44,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-14 03:53:44,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=e4ece1b6b0b8b7a536a6b6b88716f491, ASSIGN in 344 msec 2023-02-14 03:53:44,473 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:44,473 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346824473"}]},"ts":"1676346824473"} 2023-02-14 03:53:44,475 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-14 03:53:44,480 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:44,482 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 413 msec 2023-02-14 03:53:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35193] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:44,678 INFO [Listener at localhost/44029] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-14 03:53:44,678 DEBUG [Listener at localhost/44029] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-14 03:53:44,678 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,682 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-14 03:53:44,682 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,682 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-14 03:53:44,683 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,689 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,693 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,697 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:44,699 INFO [Listener at localhost/44029] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-14 03:53:44,710 INFO [Listener at localhost/44029] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-14 03:53:44,710 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(157): Stopping HMaster and jenkins-hbase9.apache.org,41249,1676346819792 server 2023-02-14 03:53:44,711 INFO [Listener at localhost/44029] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase9:35193,5,FailOnTimeoutGroup] 2023-02-14 03:53:44,711 INFO [Listener at localhost/44029] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,35193,1676346818666' ***** 2023-02-14 03:53:44,711 INFO [Listener at localhost/44029] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-14 03:53:44,712 DEBUG [M:0;jenkins-hbase9:35193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a9fa720, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:44,712 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:44,713 DEBUG [M:0;jenkins-hbase9:35193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2450ed04 to 127.0.0.1:60441 2023-02-14 03:53:44,713 DEBUG [M:0;jenkins-hbase9:35193] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:44,717 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,35193,1676346818666; all regions closed. 2023-02-14 03:53:44,717 DEBUG [M:0;jenkins-hbase9:35193] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:44,718 DEBUG [M:0;jenkins-hbase9:35193] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-14 03:53:44,718 DEBUG [M:0;jenkins-hbase9:35193] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-14 03:53:44,719 INFO [M:0;jenkins-hbase9:35193] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-14 03:53:44,719 INFO [M:0;jenkins-hbase9:35193] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-14 03:53:44,719 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346821233] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346821233,5,FailOnTimeoutGroup] 2023-02-14 03:53:44,719 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346821233] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346821233,5,FailOnTimeoutGroup] 2023-02-14 03:53:44,719 INFO [M:0;jenkins-hbase9:35193] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-02-14 03:53:44,720 DEBUG [M:0;jenkins-hbase9:35193] master.HMaster(1502): Stopping service threads 2023-02-14 03:53:44,720 INFO [M:0;jenkins-hbase9:35193] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-14 03:53:44,720 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:44,720 INFO [M:0;jenkins-hbase9:35193] procedure2.ProcedureExecutor(629): Stopping 2023-02-14 03:53:44,720 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:44,721 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-14 03:53:44,722 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:44,723 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:44,723 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:44,723 INFO [Listener at localhost/44029] hbase.LocalHBaseCluster(360): Waiting on jenkins-hbase9.apache.org,35193,1676346818666 2023-02-14 03:53:44,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:44,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:44,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:44,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:44,724 ERROR [M:0;jenkins-hbase9:35193] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-02-14 03:53:44,725 INFO [M:0;jenkins-hbase9:35193] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-14 03:53:44,725 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-14 03:53:44,726 DEBUG [M:0;jenkins-hbase9:35193] zookeeper.ZKUtil(398): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-14 03:53:44,726 WARN [M:0;jenkins-hbase9:35193] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-14 03:53:44,726 INFO [M:0;jenkins-hbase9:35193] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-14 03:53:44,726 INFO [M:0;jenkins-hbase9:35193] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-14 03:53:44,727 DEBUG [M:0;jenkins-hbase9:35193] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 03:53:44,727 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:44,727 DEBUG [M:0;jenkins-hbase9:35193] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:44,727 DEBUG [M:0;jenkins-hbase9:35193] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 03:53:44,727 DEBUG [M:0;jenkins-hbase9:35193] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:44,728 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.19 KB heapSize=76.14 KB 2023-02-14 03:53:44,798 INFO [M:0;jenkins-hbase9:35193] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.19 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0f65d904a52b4e12b17a634e574c0292 2023-02-14 03:53:44,843 DEBUG [M:0;jenkins-hbase9:35193] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0f65d904a52b4e12b17a634e574c0292 as hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0f65d904a52b4e12b17a634e574c0292 2023-02-14 03:53:44,853 INFO [M:0;jenkins-hbase9:35193] regionserver.HStore(1080): Added hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0f65d904a52b4e12b17a634e574c0292, entries=17, sequenceid=141, filesize=8.5 K 2023-02-14 03:53:44,855 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegion(2947): Finished flush of dataSize ~64.19 KB/65731, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 127ms, sequenceid=141, compaction requested=false 2023-02-14 03:53:44,858 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:44,859 DEBUG [M:0;jenkins-hbase9:35193] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676346824727Disabling compacts and flushes for region at 1676346824727Disabling writes for close at 1676346824727Obtaining lock to block concurrent updates at 1676346824728 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676346824728Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65731, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676346824738 (+10 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676346824739 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676346824742 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676346824770 (+28 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676346824774 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676346824845 (+71 ms)Finished flush of dataSize ~64.19 KB/65731, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 127ms, sequenceid=141, compaction requested=false at 1676346824855 (+10 ms)Writing region close event to WAL at 1676346824858 (+3 ms)Closed at 1676346824858 2023-02-14 03:53:44,865 INFO [M:0;jenkins-hbase9:35193] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-14 03:53:44,865 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:44,866 INFO [M:0;jenkins-hbase9:35193] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:35193 2023-02-14 03:53:44,874 DEBUG [M:0;jenkins-hbase9:35193] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,35193,1676346818666 already deleted, retry=false 2023-02-14 03:53:44,904 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:35193 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:44,904 WARN [RS-EventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:35193 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:44,905 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:35193 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 2023-02-14 03:53:44,905 DEBUG [RS-EventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:35193 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 2023-02-14 03:53:44,906 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:44,906 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:44,933 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:35193 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:44,933 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:35193 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 2023-02-14 03:53:44,934 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:44,977 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:44,978 INFO [M:0;jenkins-hbase9:35193] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,35193,1676346818666; zookeeper connection closed. 2023-02-14 03:53:44,978 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@594b5934 rejected from java.util.concurrent.ThreadPoolExecutor@29b5483e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:44,979 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:35193-0x10050f75d000000, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:44,979 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3247c252 rejected from java.util.concurrent.ThreadPoolExecutor@29b5483e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:44,980 INFO [Listener at localhost/44029] hbase.MiniHBaseCluster(508): Stopping Thread[RS:0;jenkins-hbase9:41249,5,FailOnTimeoutGroup] 2023-02-14 03:53:44,980 INFO [Listener at localhost/44029] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,41249,1676346819792' ***** 2023-02-14 03:53:44,980 INFO [Listener at localhost/44029] regionserver.HRegionServer(2310): STOPPED: Stopping rs 0 2023-02-14 03:53:44,980 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(163): Sleeping a bit 2023-02-14 03:53:45,027 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:45,107 INFO [RS:0;jenkins-hbase9:41249] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:45,107 INFO [RS:0;jenkins-hbase9:41249] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:45,107 INFO [RS:0;jenkins-hbase9:41249] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:45,107 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(3304): Received CLOSE for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:45,108 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(3304): Received CLOSE for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:45,109 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:45,109 DEBUG [RS:0;jenkins-hbase9:41249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42c21170 to 127.0.0.1:60441 2023-02-14 03:53:45,109 DEBUG [RS:0;jenkins-hbase9:41249] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:45,109 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-14 03:53:45,109 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1479): Online Regions={8fe0caa604acbdf24b118bc36deabcf4=restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4., 6babb91185b87642e80cebc762b9f388=restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.} 2023-02-14 03:53:45,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 8fe0caa604acbdf24b118bc36deabcf4, disabling compactions & flushes 2023-02-14 03:53:45,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:45,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:45,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. after waiting 0 ms 2023-02-14 03:53:45,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:45,111 DEBUG [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1505): Waiting on 6babb91185b87642e80cebc762b9f388, 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:45,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 03:53:45,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:45,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 8fe0caa604acbdf24b118bc36deabcf4: Waiting for close lock at 1676346825109Running coprocessor pre-close hooks at 1676346825109Disabling compacts and flushes for region at 1676346825109Disabling writes for close at 1676346825110 (+1 ms)Writing region close event to WAL at 1676346825110Running coprocessor post-close hooks at 1676346825117 (+7 ms)Closed at 1676346825117 2023-02-14 03:53:45,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:45,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 6babb91185b87642e80cebc762b9f388, disabling compactions & flushes 2023-02-14 03:53:45,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:45,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:45,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. after waiting 0 ms 2023-02-14 03:53:45,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:45,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 03:53:45,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:45,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 6babb91185b87642e80cebc762b9f388: Waiting for close lock at 1676346825118Running coprocessor pre-close hooks at 1676346825118Disabling compacts and flushes for region at 1676346825118Disabling writes for close at 1676346825119 (+1 ms)Writing region close event to WAL at 1676346825119Running coprocessor post-close hooks at 1676346825124 (+5 ms)Closed at 1676346825124 2023-02-14 03:53:45,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:45,128 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:45,311 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,41249,1676346819792; all regions closed. 2023-02-14 03:53:45,321 DEBUG [RS:0;jenkins-hbase9:41249] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs 2023-02-14 03:53:45,321 INFO [RS:0;jenkins-hbase9:41249] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C41249%2C1676346819792:(num 1676346821587) 2023-02-14 03:53:45,321 DEBUG [RS:0;jenkins-hbase9:41249] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:45,322 INFO [RS:0;jenkins-hbase9:41249] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:45,322 INFO [RS:0;jenkins-hbase9:41249] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:45,322 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:45,322 INFO [RS:0;jenkins-hbase9:41249] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:45,322 INFO [RS:0;jenkins-hbase9:41249] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:45,322 INFO [RS:0;jenkins-hbase9:41249] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:45,323 INFO [RS:0;jenkins-hbase9:41249] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:41249 2023-02-14 03:53:45,326 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:45,326 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@16af48b1 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:45,327 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:45,327 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:45,327 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:45,327 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:45,327 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:45,327 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a094b61 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:45,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:45,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:45,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:45,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:45,330 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 znode expired, triggering replicatorRemoved event 2023-02-14 03:53:45,330 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase9.apache.org,41249,1676346819792 znode expired, triggering replicatorRemoved event 2023-02-14 03:53:45,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:45,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:45,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:45,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:45,428 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:45,428 INFO [RS:0;jenkins-hbase9:41249] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,41249,1676346819792; zookeeper connection closed. 2023-02-14 03:53:45,428 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@727e2712 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:45,428 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d000001, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:45,428 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@772faa10 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:45,428 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5ca03624] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5ca03624 2023-02-14 03:53:45,910 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:45,936 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:46,914 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:46,939 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(2737): No master found; retry 2023-02-14 03:53:46,980 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(166): Starting HMaster and region server jenkins-hbase9.apache.org,41249,1676346819792 second time with the same port 2023-02-14 03:53:46,993 INFO [Listener at localhost/44029] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:46,993 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:46,994 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:46,994 INFO [Listener at localhost/44029] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:46,994 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:46,994 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:46,994 INFO [Listener at localhost/44029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:46,997 INFO [Listener at localhost/44029] ipc.NettyRpcServer(120): Bind to /172.31.2.10:39453 2023-02-14 03:53:46,998 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:47,000 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:47,001 INFO [Listener at localhost/44029] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39453 connecting to ZooKeeper ensemble=127.0.0.1:60441 2023-02-14 03:53:47,004 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:394530x0, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:47,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39453-0x10050f75d000009 connected 2023-02-14 03:53:47,009 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:47,009 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(162): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-14 03:53:47,010 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:47,010 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39453 2023-02-14 03:53:47,011 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39453 2023-02-14 03:53:47,011 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39453 2023-02-14 03:53:47,011 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39453 2023-02-14 03:53:47,011 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39453 2023-02-14 03:53:47,012 INFO [Listener at localhost/44029] master.HMaster(439): hbase.rootdir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83, hbase.cluster.distributed=false 2023-02-14 03:53:47,013 DEBUG [pool-218-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-14 03:53:47,014 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,016 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:47,017 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,019 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:47,020 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:47,020 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:47,020 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:47,020 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:47,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:47,021 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,39453,1676346826993 from backup master directory 2023-02-14 03:53:47,022 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,023 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:47,023 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:47,023 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,023 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegionServer(952): ClusterId : 91d4bdd4-b4a8-4072-9e0b-d4d994d6ea46 2023-02-14 03:53:47,030 INFO [Listener at localhost/44029] client.ConnectionUtils(127): regionserver/jenkins-hbase9:41249 server-side Connection retries=45 2023-02-14 03:53:47,030 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:47,030 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:47,030 INFO [Listener at localhost/44029] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:47,031 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:47,031 INFO [Listener at localhost/44029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:47,031 INFO [Listener at localhost/44029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:47,032 INFO [Listener at localhost/44029] ipc.NettyRpcServer(120): Bind to /172.31.2.10:41249 2023-02-14 03:53:47,033 INFO [Listener at localhost/44029] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:47,036 DEBUG [Listener at localhost/44029] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:47,037 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:47,038 INFO [Listener at localhost/44029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:47,039 INFO [Listener at localhost/44029] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41249 connecting to ZooKeeper ensemble=127.0.0.1:60441 2023-02-14 03:53:47,044 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:412490x0, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:47,045 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(162): regionserver:412490x0, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:47,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41249-0x10050f75d00000a connected 2023-02-14 03:53:47,047 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-14 03:53:47,048 DEBUG [Listener at localhost/44029] zookeeper.ZKUtil(164): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:47,049 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41249 2023-02-14 03:53:47,051 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41249 2023-02-14 03:53:47,051 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41249 2023-02-14 03:53:47,051 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41249 2023-02-14 03:53:47,051 DEBUG [Listener at localhost/44029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41249 2023-02-14 03:53:47,053 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:47,053 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(952): ClusterId : 91d4bdd4-b4a8-4072-9e0b-d4d994d6ea46 2023-02-14 03:53:47,054 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:47,060 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:47,060 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:47,062 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:47,063 DEBUG [RS:3;jenkins-hbase9:41249] zookeeper.ReadOnlyZKClient(139): Connect 0x5888e228 to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:47,067 DEBUG [RS:3;jenkins-hbase9:41249] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@564f0749, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:47,067 DEBUG [RS:3;jenkins-hbase9:41249] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52708958, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:47,079 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase9:41249 2023-02-14 03:53:47,079 INFO [RS:3;jenkins-hbase9:41249] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:47,079 INFO [RS:3;jenkins-hbase9:41249] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:47,079 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:47,080 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,39453,1676346826993 with isa=jenkins-hbase9.apache.org/172.31.2.10:41249, startcode=1676346827029 2023-02-14 03:53:47,080 DEBUG [RS:3;jenkins-hbase9:41249] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:47,088 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51541, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:47,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39453] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:47,104 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7a57d13e to 127.0.0.1:60441 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:47,107 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d067313, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:47,107 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:47,108 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-14 03:53:47,109 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:47,111 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 03:53:47,111 WARN [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-14 03:53:47,117 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666 to hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666-dead as it is dead 2023-02-14 03:53:47,119 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666-dead/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 2023-02-14 03:53:47,123 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666-dead/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 after 4ms 2023-02-14 03:53:47,125 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666-dead/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 to hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 2023-02-14 03:53:47,125 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,35193,1676346818666-dead 2023-02-14 03:53:47,126 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,129 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C39453%2C1676346826993, suffix=, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,39453,1676346826993, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/oldWALs, maxLogs=10 2023-02-14 03:53:47,145 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:47,146 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:47,145 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:47,149 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/WALs/jenkins-hbase9.apache.org,39453,1676346826993/jenkins-hbase9.apache.org%2C39453%2C1676346826993.1676346827130 2023-02-14 03:53:47,149 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK], DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK], DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK]] 2023-02-14 03:53:47,150 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:47,150 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:47,150 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:47,150 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:47,152 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:47,153 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-14 03:53:47,154 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-14 03:53:47,161 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0f65d904a52b4e12b17a634e574c0292 2023-02-14 03:53:47,162 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:47,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-14 03:53:47,164 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 2023-02-14 03:53:47,181 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 2023-02-14 03:53:47,183 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C35193%2C1676346818666.1676346820644 2023-02-14 03:53:47,187 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:47,189 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-14 03:53:47,190 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72831078, jitterRate=0.08526763319969177}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:47,191 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676346827150Initializing all the Stores at 1676346827151 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346827151Cleaning up temporary data from old regions at 1676346827183 (+32 ms)Cleaning up detritus from prior splits at 1676346827184 (+1 ms)Region opened successfully at 1676346827190 (+6 ms) 2023-02-14 03:53:47,191 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-14 03:53:47,192 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-14 03:53:47,192 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-14 03:53:47,192 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-14 03:53:47,193 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-14 03:53:47,206 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:47,207 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-14 03:53:47,207 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-14 03:53:47,207 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-14 03:53:47,208 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-14 03:53:47,208 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-14 03:53:47,208 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-14 03:53:47,208 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 15 msec 2023-02-14 03:53:47,209 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-14 03:53:47,209 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-14 03:53:47,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,45703,1676346819954, table=hbase:meta, region=1588230740 2023-02-14 03:53:47,211 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,39453,1676346826993 with isa=jenkins-hbase9.apache.org/172.31.2.10:41249, startcode=1676346827029 2023-02-14 03:53:47,212 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-14 03:53:47,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39453] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:47,213 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 03:53:47,213 WARN [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-14 03:53:47,213 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:47,214 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:47,215 DEBUG [RS:2;jenkins-hbase9:35309] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:47,215 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,41249,1676346819792 already deleted, retry=false 2023-02-14 03:53:47,216 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,41249,1676346819792 on jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,217 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:47701, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:47,218 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39453] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:47,224 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,41249,1676346819792, splitWal=true, meta=false 2023-02-14 03:53:47,224 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase9.apache.org,41249,1676346819792 (carryingMeta=false) jenkins-hbase9.apache.org,41249,1676346819792/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@6fb8f4fc[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-14 03:53:47,235 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-14 03:53:47,235 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-14 03:53:47,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-14 03:53:47,236 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-14 03:53:47,236 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-14 03:53:47,237 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-14 03:53:47,237 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-14 03:53:47,238 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-14 03:53:47,238 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase9.apache.org,39453,1676346826993, sessionid=0x10050f75d000009, setting cluster-up flag (Was=true) 2023-02-14 03:53:47,240 DEBUG [RS:1;jenkins-hbase9:45703] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:47,241 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-14 03:53:47,242 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,242 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53433, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:47,243 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39453] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:47,246 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-14 03:53:47,247 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:47,247 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/.hbase-snapshot/.tmp 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:47,253 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,257 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676346857257 2023-02-14 03:53:47,257 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-14 03:53:47,257 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-14 03:53:47,258 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-14 03:53:47,258 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-14 03:53:47,258 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-14 03:53:47,258 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-14 03:53:47,258 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,262 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-14 03:53:47,262 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-14 03:53:47,263 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase9.apache.org,41249,1676346819792; numProcessing=1 2023-02-14 03:53:47,263 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-14 03:53:47,263 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,41249,1676346819792, splitWal=true, meta=false 2023-02-14 03:53:47,264 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-14 03:53:47,264 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-14 03:53:47,270 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346827264,5,FailOnTimeoutGroup] 2023-02-14 03:53:47,270 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346827270,5,FailOnTimeoutGroup] 2023-02-14 03:53:47,270 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,270 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-14 03:53:47,270 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,270 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,270 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676346827017, completionTime=-1 2023-02-14 03:53:47,270 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-14 03:53:47,338 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-14 03:53:47,414 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,39453,1676346826993 with isa=jenkins-hbase9.apache.org/172.31.2.10:41249, startcode=1676346827029 2023-02-14 03:53:47,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39453] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,416 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83 2023-02-14 03:53:47,416 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39363 2023-02-14 03:53:47,416 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:47,418 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:47,418 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:47,418 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:47,418 DEBUG [RS:3;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,419 WARN [RS:3;jenkins-hbase9:41249] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:47,419 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-14 03:53:47,419 INFO [RS:3;jenkins-hbase9:41249] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:47,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,419 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,420 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,41249,1676346827029] 2023-02-14 03:53:47,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:47,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:47,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:47,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=151ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-14 03:53:47,421 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-14 03:53:47,422 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableThree' 2023-02-14 03:53:47,423 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-14 03:53:47,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:47,430 DEBUG [hconnection-0x53783886-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:47,430 DEBUG [RS:3;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,431 DEBUG [RS:3;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:47,432 DEBUG [RS:3;jenkins-hbase9:41249] zookeeper.ZKUtil(162): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:47,432 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:49026, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-14 03:53:47,433 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:47,433 INFO [RS:3;jenkins-hbase9:41249] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:47,435 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=899a4f097a47243a194517a0f9f5faf2, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35309,1676346820119, regionLocation=jenkins-hbase9.apache.org,35309,1676346820119, openSeqNum=2 2023-02-14 03:53:47,436 INFO [RS:3;jenkins-hbase9:41249] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:47,436 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=8fe0caa604acbdf24b118bc36deabcf4, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,41249,1676346819792, regionLocation=jenkins-hbase9.apache.org,41249,1676346819792, openSeqNum=2 2023-02-14 03:53:47,436 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e4ece1b6b0b8b7a536a6b6b88716f491, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35309,1676346820119, regionLocation=jenkins-hbase9.apache.org,35309,1676346820119, openSeqNum=2 2023-02-14 03:53:47,436 INFO [RS:3;jenkins-hbase9:41249] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:47,437 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=6babb91185b87642e80cebc762b9f388, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,41249,1676346819792, regionLocation=jenkins-hbase9.apache.org,41249,1676346819792, openSeqNum=2 2023-02-14 03:53:47,437 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,437 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-14 03:53:47,437 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676346887437 2023-02-14 03:53:47,437 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:47,437 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676346947437 2023-02-14 03:53:47,437 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 16 msec 2023-02-14 03:53:47,438 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,438 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,438 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:41249, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,439 DEBUG [RS:3;jenkins-hbase9:41249] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:41249, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:47,440 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,440 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,440 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,458 INFO [RS:3;jenkins-hbase9:41249] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:47,458 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41249,1676346827029-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,466 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,39453,1676346826993-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,466 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,39453,1676346826993-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,466 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,39453,1676346826993-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,466 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:39453, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,466 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:47,466 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,41249,1676346819792 had 2 regions 2023-02-14 03:53:47,467 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-14 03:53:47,469 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,41249,1676346819792, splitWal=true, meta=false, isMeta: false 2023-02-14 03:53:47,471 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792-splitting 2023-02-14 03:53:47,472 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:47,475 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:57970, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-14 03:53:47,475 WARN [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase9.apache.org,41249,1676346819792/restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4., unknown_server=jenkins-hbase9.apache.org,41249,1676346819792/restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:47,475 INFO [RS:3;jenkins-hbase9:41249] regionserver.Replication(203): jenkins-hbase9.apache.org,41249,1676346827029 started 2023-02-14 03:53:47,475 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,41249,1676346827029, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:41249, sessionid=0x10050f75d00000a 2023-02-14 03:53:47,475 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:47,475 DEBUG [RS:3;jenkins-hbase9:41249] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,475 DEBUG [RS:3;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,41249,1676346827029' 2023-02-14 03:53:47,475 DEBUG [RS:3;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:47,475 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792-splitting dir is empty, no logs to split. 2023-02-14 03:53:47,476 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase9.apache.org,41249,1676346819792 WAL count=0, meta=false 2023-02-14 03:53:47,476 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:47,476 DEBUG [RS:3;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:47,477 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:47,477 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:47,477 DEBUG [RS:3;jenkins-hbase9:41249] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,477 DEBUG [RS:3;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,41249,1676346827029' 2023-02-14 03:53:47,477 DEBUG [RS:3;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:47,478 DEBUG [RS:3;jenkins-hbase9:41249] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:47,478 DEBUG [RS:3;jenkins-hbase9:41249] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:47,478 INFO [RS:3;jenkins-hbase9:41249] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:47,478 INFO [RS:3;jenkins-hbase9:41249] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:47,480 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792-splitting dir is empty, no logs to split. 2023-02-14 03:53:47,480 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase9.apache.org,41249,1676346819792 WAL count=0, meta=false 2023-02-14 03:53:47,480 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,41249,1676346819792 WAL splitting is done? wals=0, meta=false 2023-02-14 03:53:47,482 WARN [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:35193 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:47,483 WARN [Listener at localhost/44029] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase9.apache.org/172.31.2.10:35193 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:47,483 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:35193 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:35193 2023-02-14 03:53:47,484 DEBUG [Listener at localhost/44029] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:35193 this server is in the failed servers list 2023-02-14 03:53:47,484 INFO [PEWorker-3] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase9.apache.org,41249,1676346819792 failed, ignore...File hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346819792-splitting does not exist. 2023-02-14 03:53:47,484 WARN [Listener at localhost/44029] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase9.apache.org/172.31.2.10:35193 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase9.apache.org/172.31.2.10:35193 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnSingleRSRestart(TestRetainAssignmentOnRestart.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase9.apache.org/172.31.2.10:35193 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-14 03:53:47,486 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN}] 2023-02-14 03:53:47,486 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-14 03:53:47,488 DEBUG [Listener at localhost/44029] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-14 03:53:47,491 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-14 03:53:47,491 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 0.468sec 2023-02-14 03:53:47,494 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-14 03:53:47,494 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-14 03:53:47,494 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN 2023-02-14 03:53:47,494 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-14 03:53:47,495 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,39453,1676346826993-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-14 03:53:47,495 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN 2023-02-14 03:53:47,495 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,39453,1676346826993-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-14 03:53:47,504 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:47,504 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,41249,1676346819792 2023-02-14 03:53:47,504 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,41249,1676346819792; forceNewPlan=false, retain=true 2023-02-14 03:53:47,504 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,41249,1676346819792; forceNewPlan=false, retain=true 2023-02-14 03:53:47,505 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:45460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-14 03:53:47,505 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-14 03:53:47,516 DEBUG [hconnection-0x3535bc5e-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:47,520 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33202, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41249] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Scan size: 155 connection: 172.31.2.10:33202 deadline: 1676346887520, exception=org.apache.hadoop.hbase.NotServingRegionException: restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. is not online on jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,581 INFO [RS:3;jenkins-hbase9:41249] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C41249%2C1676346827029, suffix=, logDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346827029, archiveDir=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs, maxLogs=32 2023-02-14 03:53:47,601 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK] 2023-02-14 03:53:47,601 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK] 2023-02-14 03:53:47,601 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK] 2023-02-14 03:53:47,603 INFO [RS:3;jenkins-hbase9:41249] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,41249,1676346827029/jenkins-hbase9.apache.org%2C41249%2C1676346827029.1676346827582 2023-02-14 03:53:47,603 DEBUG [RS:3;jenkins-hbase9:41249] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39919,DS-e0682349-ed5b-464f-83b9-71b868c7df59,DISK], DatanodeInfoWithStorage[127.0.0.1:46195,DS-eff5587b-7bb3-43e4-89ae-46014e96fc7f,DISK], DatanodeInfoWithStorage[127.0.0.1:44313,DS-58d70f9a-812b-4b75-bec3-2e8f525ffd5a,DISK]] 2023-02-14 03:53:47,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41249] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Scan size: 155 connection: 172.31.2.10:33202 deadline: 1676346887516, exception=org.apache.hadoop.hbase.NotServingRegionException: restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. is not online on jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,655 INFO [jenkins-hbase9:39453] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-14 03:53:47,657 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=6babb91185b87642e80cebc762b9f388, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,657 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=8fe0caa604acbdf24b118bc36deabcf4, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346827657"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346827657"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346827657"}]},"ts":"1676346827657"} 2023-02-14 03:53:47,657 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346827657"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346827657"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346827657"}]},"ts":"1676346827657"} 2023-02-14 03:53:47,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 6babb91185b87642e80cebc762b9f388, server=jenkins-hbase9.apache.org,41249,1676346827029}] 2023-02-14 03:53:47,660 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE; OpenRegionProcedure 8fe0caa604acbdf24b118bc36deabcf4, server=jenkins-hbase9.apache.org,41249,1676346827029}] 2023-02-14 03:53:47,812 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,813 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:47,814 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33214, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=AdminService 2023-02-14 03:53:47,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:47,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8fe0caa604acbdf24b118bc36deabcf4, NAME => 'restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:47,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:47,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(7890): checking encryption for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(7893): checking classloading for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,821 INFO [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,822 DEBUG [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/family 2023-02-14 03:53:47,822 DEBUG [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/family 2023-02-14 03:53:47,823 INFO [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fe0caa604acbdf24b118bc36deabcf4 columnFamilyName family 2023-02-14 03:53:47,824 INFO [StoreOpener-8fe0caa604acbdf24b118bc36deabcf4-1] regionserver.HStore(310): Store=8fe0caa604acbdf24b118bc36deabcf4/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:47,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1054): writing seq id for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:47,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1071): Opened 8fe0caa604acbdf24b118bc36deabcf4; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74802560, jitterRate=0.11464500427246094}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:47,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(964): Region open journal for 8fe0caa604acbdf24b118bc36deabcf4: Running coprocessor pre-open hook at 1676346827819Writing region info on filesystem at 1676346827819Initializing all the Stores at 1676346827820 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346827820Cleaning up temporary data from old regions at 1676346827827 (+7 ms)Cleaning up detritus from prior splits at 1676346827828 (+1 ms)Running coprocessor post-open hooks at 1676346827832 (+4 ms)Region opened successfully at 1676346827832 2023-02-14 03:53:47,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4., pid=22, masterSystemTime=1676346827812 2023-02-14 03:53:47,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41249] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Scan size: 155 connection: 172.31.2.10:33202 deadline: 1676346887516, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. is opening on jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:47,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:47,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:47,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6babb91185b87642e80cebc762b9f388, NAME => 'restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:47,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:47,838 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=8fe0caa604acbdf24b118bc36deabcf4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(7890): checking encryption for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(7893): checking classloading for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,839 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346827838"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346827838"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346827838"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346827838"}]},"ts":"1676346827838"} 2023-02-14 03:53:47,841 INFO [StoreOpener-6babb91185b87642e80cebc762b9f388-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,842 DEBUG [StoreOpener-6babb91185b87642e80cebc762b9f388-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/family 2023-02-14 03:53:47,842 DEBUG [StoreOpener-6babb91185b87642e80cebc762b9f388-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/family 2023-02-14 03:53:47,843 INFO [StoreOpener-6babb91185b87642e80cebc762b9f388-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6babb91185b87642e80cebc762b9f388 columnFamilyName family 2023-02-14 03:53:47,843 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-02-14 03:53:47,843 INFO [StoreOpener-6babb91185b87642e80cebc762b9f388-1] regionserver.HStore(310): Store=6babb91185b87642e80cebc762b9f388/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:47,844 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; OpenRegionProcedure 8fe0caa604acbdf24b118bc36deabcf4, server=jenkins-hbase9.apache.org,41249,1676346827029 in 181 msec 2023-02-14 03:53:47,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,846 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=8fe0caa604acbdf24b118bc36deabcf4, ASSIGN in 358 msec 2023-02-14 03:53:47,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1054): writing seq id for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:47,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1071): Opened 6babb91185b87642e80cebc762b9f388; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67842619, jitterRate=0.010933801531791687}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:47,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(964): Region open journal for 6babb91185b87642e80cebc762b9f388: Running coprocessor pre-open hook at 1676346827839Writing region info on filesystem at 1676346827839Initializing all the Stores at 1676346827840 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346827840Cleaning up temporary data from old regions at 1676346827847 (+7 ms)Cleaning up detritus from prior splits at 1676346827848 (+1 ms)Running coprocessor post-open hooks at 1676346827851 (+3 ms)Region opened successfully at 1676346827851 2023-02-14 03:53:47,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388., pid=21, masterSystemTime=1676346827812 2023-02-14 03:53:47,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:47,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:41249-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:47,854 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=6babb91185b87642e80cebc762b9f388, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:47,855 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346827854"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346827854"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346827854"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346827854"}]},"ts":"1676346827854"} 2023-02-14 03:53:47,859 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-02-14 03:53:47,859 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 6babb91185b87642e80cebc762b9f388, server=jenkins-hbase9.apache.org,41249,1676346827029 in 198 msec 2023-02-14 03:53:47,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=18 2023-02-14 03:53:47,862 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,41249,1676346819792 after splitting done 2023-02-14 03:53:47,862 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=6babb91185b87642e80cebc762b9f388, ASSIGN in 373 msec 2023-02-14 03:53:47,862 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase9.apache.org,41249,1676346819792 from processing; numProcessing=0 2023-02-14 03:53:47,864 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,41249,1676346819792, splitWal=true, meta=false in 644 msec 2023-02-14 03:53:48,144 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:48,152 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:48,159 DEBUG [hconnection-0x3535bc5e-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:48,161 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:57974, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:48,163 INFO [Listener at localhost/44029] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:48,163 INFO [Listener at localhost/44029] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 8fe0caa604acbdf24b118bc36deabcf4, NAME => 'restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,41249,1676346819792, currentServer=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => e4ece1b6b0b8b7a536a6b6b88716f491, NAME => 'restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,35309,1676346820119, currentServer=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 6babb91185b87642e80cebc762b9f388, NAME => 'restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,41249,1676346819792, currentServer=jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 899a4f097a47243a194517a0f9f5faf2, NAME => 'hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,35309,1676346820119, currentServer=jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-14 03:53:48,166 INFO [Listener at localhost/44029] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-14 03:53:48,167 DEBUG [Listener at localhost/44029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x106f9e57 to 127.0.0.1:60441 2023-02-14 03:53:48,167 DEBUG [Listener at localhost/44029] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,167 DEBUG [Listener at localhost/44029] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-14 03:53:48,167 DEBUG [Listener at localhost/44029] util.JVMClusterUtil(257): Found active master hash=1986254219, stopped=false 2023-02-14 03:53:48,168 INFO [Listener at localhost/44029] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:48,171 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:48,171 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:48,171 INFO [Listener at localhost/44029] procedure2.ProcedureExecutor(629): Stopping 2023-02-14 03:53:48,171 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:48,171 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:48,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:48,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:48,173 DEBUG [Listener at localhost/44029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a57d13e to 127.0.0.1:60441 2023-02-14 03:53:48,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:48,171 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:48,173 DEBUG [Listener at localhost/44029] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:48,173 INFO [Listener at localhost/44029] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,45703,1676346819954' ***** 2023-02-14 03:53:48,173 INFO [Listener at localhost/44029] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:53:48,174 INFO [Listener at localhost/44029] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,35309,1676346820119' ***** 2023-02-14 03:53:48,174 INFO [Listener at localhost/44029] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:53:48,174 INFO [RS:1;jenkins-hbase9:45703] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:48,174 INFO [Listener at localhost/44029] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,41249,1676346827029' ***** 2023-02-14 03:53:48,174 INFO [RS:2;jenkins-hbase9:35309] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:48,174 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:48,174 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:48,174 INFO [RS:2;jenkins-hbase9:35309] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:48,174 INFO [RS:1;jenkins-hbase9:45703] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:48,174 INFO [Listener at localhost/44029] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:53:48,175 INFO [RS:1;jenkins-hbase9:45703] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:48,175 INFO [RS:2;jenkins-hbase9:35309] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:48,175 INFO [RS:3;jenkins-hbase9:41249] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:48,175 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:48,175 INFO [RS:3;jenkins-hbase9:41249] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:48,175 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:48,175 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(3304): Received CLOSE for 899a4f097a47243a194517a0f9f5faf2 2023-02-14 03:53:48,176 INFO [RS:3;jenkins-hbase9:41249] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:48,176 DEBUG [RS:1;jenkins-hbase9:45703] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x294828c6 to 127.0.0.1:60441 2023-02-14 03:53:48,176 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(3304): Received CLOSE for 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:48,176 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(3304): Received CLOSE for e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:48,176 DEBUG [RS:1;jenkins-hbase9:45703] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,176 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:48,177 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(3304): Received CLOSE for 6babb91185b87642e80cebc762b9f388 2023-02-14 03:53:48,177 INFO [RS:1;jenkins-hbase9:45703] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:48,177 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:48,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 899a4f097a47243a194517a0f9f5faf2, disabling compactions & flushes 2023-02-14 03:53:48,177 DEBUG [RS:3;jenkins-hbase9:41249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5888e228 to 127.0.0.1:60441 2023-02-14 03:53:48,177 INFO [RS:1;jenkins-hbase9:45703] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:48,177 DEBUG [RS:2;jenkins-hbase9:35309] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33bf36bb to 127.0.0.1:60441 2023-02-14 03:53:48,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1603): Closing 8fe0caa604acbdf24b118bc36deabcf4, disabling compactions & flushes 2023-02-14 03:53:48,178 DEBUG [RS:2;jenkins-hbase9:35309] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,178 INFO [RS:1;jenkins-hbase9:45703] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:48,178 DEBUG [RS:3;jenkins-hbase9:41249] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:48,178 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-14 03:53:48,178 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-14 03:53:48,178 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-14 03:53:48,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:48,178 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1479): Online Regions={899a4f097a47243a194517a0f9f5faf2=hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2., e4ece1b6b0b8b7a536a6b6b88716f491=restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491.} 2023-02-14 03:53:48,178 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1479): Online Regions={8fe0caa604acbdf24b118bc36deabcf4=restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4., 6babb91185b87642e80cebc762b9f388=restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388.} 2023-02-14 03:53:48,178 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-14 03:53:48,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:48,178 DEBUG [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1505): Waiting on 6babb91185b87642e80cebc762b9f388, 8fe0caa604acbdf24b118bc36deabcf4 2023-02-14 03:53:48,178 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-14 03:53:48,178 DEBUG [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1505): Waiting on 899a4f097a47243a194517a0f9f5faf2, e4ece1b6b0b8b7a536a6b6b88716f491 2023-02-14 03:53:48,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:48,179 DEBUG [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-14 03:53:48,179 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 03:53:48,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. after waiting 0 ms 2023-02-14 03:53:48,179 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 03:53:48,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:48,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. after waiting 0 ms 2023-02-14 03:53:48,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 899a4f097a47243a194517a0f9f5faf2 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-14 03:53:48,179 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 03:53:48,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:48,179 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 03:53:48,180 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 03:53:48,180 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=7.04 KB heapSize=12.50 KB 2023-02-14 03:53:48,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableOne/8fe0caa604acbdf24b118bc36deabcf4/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-14 03:53:48,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1837): Closed restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:48,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1557): Region close journal for 8fe0caa604acbdf24b118bc36deabcf4: Waiting for close lock at 1676346828177Running coprocessor pre-close hooks at 1676346828177Disabling compacts and flushes for region at 1676346828177Disabling writes for close at 1676346828179 (+2 ms)Writing region close event to WAL at 1676346828180 (+1 ms)Running coprocessor post-close hooks at 1676346828186 (+6 ms)Closed at 1676346828186 2023-02-14 03:53:48,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676346822810.8fe0caa604acbdf24b118bc36deabcf4. 2023-02-14 03:53:48,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1603): Closing 6babb91185b87642e80cebc762b9f388, disabling compactions & flushes 2023-02-14 03:53:48,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:48,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:48,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. after waiting 0 ms 2023-02-14 03:53:48,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:48,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableTwo/6babb91185b87642e80cebc762b9f388/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-14 03:53:48,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:48,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] regionserver.HRegion(1557): Region close journal for 6babb91185b87642e80cebc762b9f388: Waiting for close lock at 1676346828186Running coprocessor pre-close hooks at 1676346828186Disabling compacts and flushes for region at 1676346828186Disabling writes for close at 1676346828187 (+1 ms)Writing region close event to WAL at 1676346828187Running coprocessor post-close hooks at 1676346828196 (+9 ms)Closed at 1676346828196 2023-02-14 03:53:48,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:41249-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676346823450.6babb91185b87642e80cebc762b9f388. 2023-02-14 03:53:48,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/.tmp/info/6949bf595a174c418b7395a9a4cf78e3 2023-02-14 03:53:48,212 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.67 KB at sequenceid=28 (bloomFilter=false), to=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/.tmp/info/f50f906b63b64ea18d4f8309a47c5cc1 2023-02-14 03:53:48,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/.tmp/info/6949bf595a174c418b7395a9a4cf78e3 as hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/info/6949bf595a174c418b7395a9a4cf78e3 2023-02-14 03:53:48,230 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:48,230 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:48,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/info/6949bf595a174c418b7395a9a4cf78e3, entries=2, sequenceid=6, filesize=4.8 K 2023-02-14 03:53:48,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 899a4f097a47243a194517a0f9f5faf2 in 55ms, sequenceid=6, compaction requested=false 2023-02-14 03:53:48,266 INFO [regionserver/jenkins-hbase9:41249.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:48,278 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=28 (bloomFilter=false), to=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/.tmp/table/19ca1c8241fa4506a984b8b55ab6db31 2023-02-14 03:53:48,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/namespace/899a4f097a47243a194517a0f9f5faf2/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-14 03:53:48,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:48,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 899a4f097a47243a194517a0f9f5faf2: Waiting for close lock at 1676346828176Running coprocessor pre-close hooks at 1676346828176Disabling compacts and flushes for region at 1676346828176Disabling writes for close at 1676346828179 (+3 ms)Obtaining lock to block concurrent updates at 1676346828179Preparing flush snapshotting stores in 899a4f097a47243a194517a0f9f5faf2 at 1676346828179Finished memstore snapshotting hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676346828180 (+1 ms)Flushing stores of hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. at 1676346828181 (+1 ms)Flushing 899a4f097a47243a194517a0f9f5faf2/info: creating writer at 1676346828181Flushing 899a4f097a47243a194517a0f9f5faf2/info: appending metadata at 1676346828187 (+6 ms)Flushing 899a4f097a47243a194517a0f9f5faf2/info: closing flushed file at 1676346828187Flushing 899a4f097a47243a194517a0f9f5faf2/info: reopening flushed file at 1676346828224 (+37 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 899a4f097a47243a194517a0f9f5faf2 in 55ms, sequenceid=6, compaction requested=false at 1676346828234 (+10 ms)Writing region close event to WAL at 1676346828270 (+36 ms)Running coprocessor post-close hooks at 1676346828280 (+10 ms)Closed at 1676346828281 (+1 ms) 2023-02-14 03:53:48,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676346822013.899a4f097a47243a194517a0f9f5faf2. 2023-02-14 03:53:48,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing e4ece1b6b0b8b7a536a6b6b88716f491, disabling compactions & flushes 2023-02-14 03:53:48,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:48,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:48,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. after waiting 0 ms 2023-02-14 03:53:48,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:48,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/default/restartTableThree/e4ece1b6b0b8b7a536a6b6b88716f491/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 03:53:48,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:48,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for e4ece1b6b0b8b7a536a6b6b88716f491: Waiting for close lock at 1676346828281Running coprocessor pre-close hooks at 1676346828281Disabling compacts and flushes for region at 1676346828281Disabling writes for close at 1676346828282 (+1 ms)Writing region close event to WAL at 1676346828282Running coprocessor post-close hooks at 1676346828287 (+5 ms)Closed at 1676346828287 2023-02-14 03:53:48,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676346824067.e4ece1b6b0b8b7a536a6b6b88716f491. 2023-02-14 03:53:48,288 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/.tmp/info/f50f906b63b64ea18d4f8309a47c5cc1 as hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/info/f50f906b63b64ea18d4f8309a47c5cc1 2023-02-14 03:53:48,295 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/info/f50f906b63b64ea18d4f8309a47c5cc1, entries=48, sequenceid=28, filesize=10.4 K 2023-02-14 03:53:48,296 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/.tmp/table/19ca1c8241fa4506a984b8b55ab6db31 as hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/table/19ca1c8241fa4506a984b8b55ab6db31 2023-02-14 03:53:48,303 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/table/19ca1c8241fa4506a984b8b55ab6db31, entries=8, sequenceid=28, filesize=5.0 K 2023-02-14 03:53:48,304 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~7.04 KB/7210, heapSize ~12.22 KB/12512, currentSize=0 B/0 for 1588230740 in 124ms, sequenceid=28, compaction requested=false 2023-02-14 03:53:48,312 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/data/hbase/meta/1588230740/recovered.edits/31.seqid, newMaxSeqId=31, maxSeqId=1 2023-02-14 03:53:48,313 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-14 03:53:48,314 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 03:53:48,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676346828178Running coprocessor pre-close hooks at 1676346828178Disabling compacts and flushes for region at 1676346828178Disabling writes for close at 1676346828180 (+2 ms)Obtaining lock to block concurrent updates at 1676346828180Preparing flush snapshotting stores in 1588230740 at 1676346828180Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=7210, getHeapSize=12752, getOffHeapSize=0, getCellsCount=64 at 1676346828180Flushing stores of hbase:meta,,1.1588230740 at 1676346828181 (+1 ms)Flushing 1588230740/info: creating writer at 1676346828181Flushing 1588230740/info: appending metadata at 1676346828188 (+7 ms)Flushing 1588230740/info: closing flushed file at 1676346828188Flushing 1588230740/table: creating writer at 1676346828222 (+34 ms)Flushing 1588230740/table: appending metadata at 1676346828226 (+4 ms)Flushing 1588230740/table: closing flushed file at 1676346828226Flushing 1588230740/info: reopening flushed file at 1676346828289 (+63 ms)Flushing 1588230740/table: reopening flushed file at 1676346828297 (+8 ms)Finished flush of dataSize ~7.04 KB/7210, heapSize ~12.22 KB/12512, currentSize=0 B/0 for 1588230740 in 124ms, sequenceid=28, compaction requested=false at 1676346828304 (+7 ms)Writing region close event to WAL at 1676346828307 (+3 ms)Running coprocessor post-close hooks at 1676346828313 (+6 ms)Closed at 1676346828314 (+1 ms) 2023-02-14 03:53:48,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-14 03:53:48,379 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,41249,1676346827029; all regions closed. 2023-02-14 03:53:48,379 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,45703,1676346819954; all regions closed. 2023-02-14 03:53:48,379 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,35309,1676346820119; all regions closed. 2023-02-14 03:53:48,386 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,45703,1676346819954/jenkins-hbase9.apache.org%2C45703%2C1676346819954.meta.1676346821760.meta not finished, retry = 0 2023-02-14 03:53:48,387 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/WALs/jenkins-hbase9.apache.org,35309,1676346820119/jenkins-hbase9.apache.org%2C35309%2C1676346820119.1676346821588 not finished, retry = 0 2023-02-14 03:53:48,389 DEBUG [RS:3;jenkins-hbase9:41249] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs 2023-02-14 03:53:48,389 INFO [RS:3;jenkins-hbase9:41249] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C41249%2C1676346827029:(num 1676346827582) 2023-02-14 03:53:48,389 DEBUG [RS:3;jenkins-hbase9:41249] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,389 INFO [RS:3;jenkins-hbase9:41249] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:48,389 INFO [RS:3;jenkins-hbase9:41249] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:41249 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:48,389 INFO [RS:3;jenkins-hbase9:41249] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:48,389 INFO [regionserver/jenkins-hbase9:41249.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:48,390 INFO [RS:3;jenkins-hbase9:41249] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:48,390 INFO [RS:3;jenkins-hbase9:41249] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:48,390 INFO [RS:3;jenkins-hbase9:41249] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:41249 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,41249,1676346827029 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:48,396 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3cc45a04 rejected from java.util.concurrent.ThreadPoolExecutor@38fbd944[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,396 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:48,396 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@52c4dfa6 rejected from java.util.concurrent.ThreadPoolExecutor@38fbd944[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,397 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,41249,1676346827029] 2023-02-14 03:53:48,397 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,41249,1676346827029; numProcessing=1 2023-02-14 03:53:48,398 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,41249,1676346827029 already deleted, retry=false 2023-02-14 03:53:48,398 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,41249,1676346827029 expired; onlineServers=2 2023-02-14 03:53:48,424 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-14 03:53:48,424 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-14 03:53:48,476 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-14 03:53:48,476 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-14 03:53:48,490 DEBUG [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs 2023-02-14 03:53:48,490 INFO [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C45703%2C1676346819954.meta:.meta(num 1676346821760) 2023-02-14 03:53:48,490 DEBUG [RS:2;jenkins-hbase9:35309] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs 2023-02-14 03:53:48,490 INFO [RS:2;jenkins-hbase9:35309] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C35309%2C1676346820119:(num 1676346821588) 2023-02-14 03:53:48,490 DEBUG [RS:2;jenkins-hbase9:35309] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,490 INFO [RS:2;jenkins-hbase9:35309] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:48,491 INFO [RS:2;jenkins-hbase9:35309] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:48,491 INFO [RS:2;jenkins-hbase9:35309] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:48,491 INFO [RS:2;jenkins-hbase9:35309] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:48,491 INFO [RS:2;jenkins-hbase9:35309] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:48,491 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:48,492 INFO [RS:2;jenkins-hbase9:35309] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:35309 2023-02-14 03:53:48,495 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:48,496 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:48,496 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,35309,1676346820119 2023-02-14 03:53:48,499 DEBUG [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/oldWALs 2023-02-14 03:53:48,499 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,35309,1676346820119] 2023-02-14 03:53:48,499 INFO [RS:1;jenkins-hbase9:45703] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C45703%2C1676346819954:(num 1676346821588) 2023-02-14 03:53:48,499 DEBUG [RS:1;jenkins-hbase9:45703] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,499 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,35309,1676346820119; numProcessing=2 2023-02-14 03:53:48,499 INFO [RS:1;jenkins-hbase9:45703] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:48,500 INFO [RS:1;jenkins-hbase9:45703] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:48,500 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:48,500 INFO [RS:1;jenkins-hbase9:45703] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:45703 2023-02-14 03:53:48,501 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,35309,1676346820119 already deleted, retry=false 2023-02-14 03:53:48,501 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,35309,1676346820119 expired; onlineServers=1 2023-02-14 03:53:48,502 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,45703,1676346819954 2023-02-14 03:53:48,502 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:48,502 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14193d45 rejected from java.util.concurrent.ThreadPoolExecutor@1026bb1e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,504 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,45703,1676346819954] 2023-02-14 03:53:48,504 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,45703,1676346819954; numProcessing=3 2023-02-14 03:53:48,506 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,45703,1676346819954 already deleted, retry=false 2023-02-14 03:53:48,506 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,45703,1676346819954 expired; onlineServers=0 2023-02-14 03:53:48,506 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,39453,1676346826993' ***** 2023-02-14 03:53:48,506 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-14 03:53:48,507 DEBUG [M:0;jenkins-hbase9:39453] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@725c4f9f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:48,507 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,39453,1676346826993 2023-02-14 03:53:48,507 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,39453,1676346826993; all regions closed. 2023-02-14 03:53:48,507 DEBUG [M:0;jenkins-hbase9:39453] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:48,508 DEBUG [M:0;jenkins-hbase9:39453] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-14 03:53:48,508 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-14 03:53:48,508 DEBUG [M:0;jenkins-hbase9:39453] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-14 03:53:48,508 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346827264] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346827264,5,FailOnTimeoutGroup] 2023-02-14 03:53:48,508 INFO [M:0;jenkins-hbase9:39453] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-14 03:53:48,508 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346827270] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346827270,5,FailOnTimeoutGroup] 2023-02-14 03:53:48,509 INFO [M:0;jenkins-hbase9:39453] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-14 03:53:48,509 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:48,509 INFO [M:0;jenkins-hbase9:39453] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-02-14 03:53:48,510 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:48,510 DEBUG [M:0;jenkins-hbase9:39453] master.HMaster(1502): Stopping service threads 2023-02-14 03:53:48,510 INFO [M:0;jenkins-hbase9:39453] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-14 03:53:48,510 ERROR [M:0;jenkins-hbase9:39453] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-14 03:53:48,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:48,510 INFO [M:0;jenkins-hbase9:39453] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-14 03:53:48,510 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-14 03:53:48,511 DEBUG [M:0;jenkins-hbase9:39453] zookeeper.ZKUtil(398): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-14 03:53:48,511 WARN [M:0;jenkins-hbase9:39453] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-14 03:53:48,511 INFO [M:0;jenkins-hbase9:39453] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-14 03:53:48,511 INFO [M:0;jenkins-hbase9:39453] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-14 03:53:48,512 DEBUG [M:0;jenkins-hbase9:39453] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 03:53:48,512 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:48,512 DEBUG [M:0;jenkins-hbase9:39453] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:48,512 DEBUG [M:0;jenkins-hbase9:39453] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 03:53:48,512 DEBUG [M:0;jenkins-hbase9:39453] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:48,512 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=14.68 KB heapSize=18.26 KB 2023-02-14 03:53:48,531 INFO [M:0;jenkins-hbase9:39453] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=14.68 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e0babffbe4864264969b269b752d776c 2023-02-14 03:53:48,537 DEBUG [M:0;jenkins-hbase9:39453] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e0babffbe4864264969b269b752d776c as hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e0babffbe4864264969b269b752d776c 2023-02-14 03:53:48,544 INFO [M:0;jenkins-hbase9:39453] regionserver.HStore(1080): Added hdfs://localhost:39363/user/jenkins/test-data/2a85037a-bef9-97d2-a664-a82a79216f83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e0babffbe4864264969b269b752d776c, entries=5, sequenceid=182, filesize=5.3 K 2023-02-14 03:53:48,545 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegion(2947): Finished flush of dataSize ~14.68 KB/15031, heapSize ~18.24 KB/18680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 33ms, sequenceid=182, compaction requested=false 2023-02-14 03:53:48,547 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:48,547 DEBUG [M:0;jenkins-hbase9:39453] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676346828511Disabling compacts and flushes for region at 1676346828511Disabling writes for close at 1676346828512 (+1 ms)Obtaining lock to block concurrent updates at 1676346828512Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676346828512Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=15031, getHeapSize=18680, getOffHeapSize=0, getCellsCount=45 at 1676346828512Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676346828513 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676346828514 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676346828517 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676346828517Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676346828538 (+21 ms)Finished flush of dataSize ~14.68 KB/15031, heapSize ~18.24 KB/18680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 33ms, sequenceid=182, compaction requested=false at 1676346828545 (+7 ms)Writing region close event to WAL at 1676346828547 (+2 ms)Closed at 1676346828547 2023-02-14 03:53:48,551 INFO [M:0;jenkins-hbase9:39453] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-14 03:53:48,551 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:48,553 INFO [M:0;jenkins-hbase9:39453] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:39453 2023-02-14 03:53:48,555 DEBUG [M:0;jenkins-hbase9:39453] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,39453,1676346826993 already deleted, retry=false 2023-02-14 03:53:48,771 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:48,772 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@782f6bcc rejected from java.util.concurrent.ThreadPoolExecutor@7c10f011[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,771 INFO [M:0;jenkins-hbase9:39453] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,39453,1676346826993; zookeeper connection closed. 2023-02-14 03:53:48,772 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): master:39453-0x10050f75d000009, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:48,773 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b13d627 rejected from java.util.concurrent.ThreadPoolExecutor@7c10f011[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,872 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:48,872 INFO [RS:1;jenkins-hbase9:45703] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,45703,1676346819954; zookeeper connection closed. 2023-02-14 03:53:48,872 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3915c014 rejected from java.util.concurrent.ThreadPoolExecutor@1026bb1e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,872 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:45703-0x10050f75d000002, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:48,872 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2eb063dc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2eb063dc 2023-02-14 03:53:48,872 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41e088b4 rejected from java.util.concurrent.ThreadPoolExecutor@1026bb1e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,972 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:48,972 INFO [RS:2;jenkins-hbase9:35309] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,35309,1676346820119; zookeeper connection closed. 2023-02-14 03:53:48,972 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7306df96 rejected from java.util.concurrent.ThreadPoolExecutor@7739552e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:48,973 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@670a049b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@670a049b 2023-02-14 03:53:48,973 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:35309-0x10050f75d000003, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:48,973 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@46fc2e7a rejected from java.util.concurrent.ThreadPoolExecutor@7739552e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:49,072 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:49,072 INFO [RS:3;jenkins-hbase9:41249] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,41249,1676346827029; zookeeper connection closed. 2023-02-14 03:53:49,073 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2a197aa4 rejected from java.util.concurrent.ThreadPoolExecutor@38fbd944[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:49,073 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@52bcd1c5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@52bcd1c5 2023-02-14 03:53:49,073 DEBUG [Listener at localhost/44029-EventThread] zookeeper.ZKWatcher(600): regionserver:41249-0x10050f75d00000a, quorum=127.0.0.1:60441, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:49,073 ERROR [Listener at localhost/44029-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@691b6a83 rejected from java.util.concurrent.ThreadPoolExecutor@38fbd944[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:49,073 INFO [Listener at localhost/44029] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 4 regionserver(s) complete 2023-02-14 03:53:49,074 WARN [Listener at localhost/44029] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 03:53:49,080 INFO [Listener at localhost/44029] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:53:49,185 WARN [BP-1967131781-172.31.2.10-1676346815259 heartbeating to localhost/127.0.0.1:39363] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 03:53:49,185 WARN [BP-1967131781-172.31.2.10-1676346815259 heartbeating to localhost/127.0.0.1:39363] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1967131781-172.31.2.10-1676346815259 (Datanode Uuid e010bf8c-ce27-46b0-b9ee-e8cb58902037) service to localhost/127.0.0.1:39363 2023-02-14 03:53:49,187 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/dfs/data/data5/current/BP-1967131781-172.31.2.10-1676346815259] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:53:49,187 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/dfs/data/data6/current/BP-1967131781-172.31.2.10-1676346815259] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:53:49,189 WARN [Listener at localhost/44029] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 03:53:49,191 INFO [Listener at localhost/44029] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:53:49,294 WARN [BP-1967131781-172.31.2.10-1676346815259 heartbeating to localhost/127.0.0.1:39363] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 03:53:49,294 WARN [BP-1967131781-172.31.2.10-1676346815259 heartbeating to localhost/127.0.0.1:39363] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1967131781-172.31.2.10-1676346815259 (Datanode Uuid 3ce420cf-03fc-4f40-a1ac-24a87b00d91a) service to localhost/127.0.0.1:39363 2023-02-14 03:53:49,295 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/dfs/data/data3/current/BP-1967131781-172.31.2.10-1676346815259] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:53:49,295 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/dfs/data/data4/current/BP-1967131781-172.31.2.10-1676346815259] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:53:49,296 WARN [Listener at localhost/44029] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 03:53:49,298 INFO [Listener at localhost/44029] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:53:49,401 WARN [BP-1967131781-172.31.2.10-1676346815259 heartbeating to localhost/127.0.0.1:39363] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 03:53:49,401 WARN [BP-1967131781-172.31.2.10-1676346815259 heartbeating to localhost/127.0.0.1:39363] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1967131781-172.31.2.10-1676346815259 (Datanode Uuid 5b6281ff-3c02-403e-8714-a73ce7a86e3f) service to localhost/127.0.0.1:39363 2023-02-14 03:53:49,402 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/dfs/data/data1/current/BP-1967131781-172.31.2.10-1676346815259] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:53:49,403 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/cluster_1e1d5208-638a-85f8-aa9c-023364e5bf38/dfs/data/data2/current/BP-1967131781-172.31.2.10-1676346815259] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:53:49,440 INFO [Listener at localhost/44029] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:53:49,560 INFO [Listener at localhost/44029] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-14 03:53:49,621 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-14 03:53:49,636 INFO [Listener at localhost/44029] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=101 (was 7) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: master/jenkins-hbase9:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: Listener at localhost/44029 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:39363 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:39363 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@28227280 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:39363 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:39363 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:39363 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase9:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:39363 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: regionserver/jenkins-hbase9:41249.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: ForkJoinPool-2-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:39363 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=523 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=347 (was 365), ProcessCount=171 (was 171), AvailableMemoryMB=3338 (was 4238) 2023-02-14 03:53:49,647 INFO [Listener at localhost/44029] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=101, OpenFileDescriptor=523, MaxFileDescriptor=60000, SystemLoadAverage=347, ProcessCount=171, AvailableMemoryMB=3337 2023-02-14 03:53:49,662 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/hadoop.log.dir so I do NOT create it in target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed 2023-02-14 03:53:49,663 WARN [Listener at localhost/44029] hbase.HBaseTestingUtility(449): hadoop.log.dir property value differs in configuration and system: Configuration=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/../logs while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/hadoop.log.dir Erasing configuration value by system value. 2023-02-14 03:53:49,663 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/hadoop.tmp.dir so I do NOT create it in target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed 2023-02-14 03:53:49,663 WARN [Listener at localhost/44029] hbase.HBaseTestingUtility(449): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-jenkins while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4cd3c6d0-b1db-df4c-202d-c8b37c42cb1d/hadoop.tmp.dir Erasing configuration value by system value. 2023-02-14 03:53:49,663 DEBUG [Listener at localhost/44029] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed 2023-02-14 03:53:49,663 INFO [Listener at localhost/44029] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-14 03:53:49,663 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-14 03:53:49,663 INFO [Listener at localhost/44029] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd, deleteOnExit=true 2023-02-14 03:53:49,663 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-14 03:53:49,664 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/test.cache.data in system properties and HBase conf 2023-02-14 03:53:49,664 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/hadoop.tmp.dir in system properties and HBase conf 2023-02-14 03:53:49,664 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/hadoop.log.dir in system properties and HBase conf 2023-02-14 03:53:49,664 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-14 03:53:49,665 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-14 03:53:49,665 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-14 03:53:49,665 DEBUG [Listener at localhost/44029] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-14 03:53:49,665 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-14 03:53:49,665 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-14 03:53:49,666 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-14 03:53:49,667 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/nfs.dump.dir in system properties and HBase conf 2023-02-14 03:53:49,667 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/java.io.tmpdir in system properties and HBase conf 2023-02-14 03:53:49,667 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-14 03:53:49,667 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-14 03:53:49,667 INFO [Listener at localhost/44029] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-14 03:53:49,672 WARN [Listener at localhost/44029] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-14 03:53:49,672 WARN [Listener at localhost/44029] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-14 03:53:49,716 WARN [Listener at localhost/44029] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:49,719 INFO [Listener at localhost/44029] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:49,723 INFO [Listener at localhost/44029] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/java.io.tmpdir/Jetty_localhost_43321_hdfs____.3k23k3/webapp 2023-02-14 03:53:49,827 INFO [Listener at localhost/44029] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43321 2023-02-14 03:53:49,832 WARN [Listener at localhost/44029] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-14 03:53:49,832 WARN [Listener at localhost/44029] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-14 03:53:49,879 WARN [Listener at localhost/42255] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:49,895 WARN [Listener at localhost/42255] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 03:53:49,898 WARN [Listener at localhost/42255] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:49,900 INFO [Listener at localhost/42255] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:49,908 INFO [Listener at localhost/42255] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/java.io.tmpdir/Jetty_localhost_46229_datanode____mzch3y/webapp 2023-02-14 03:53:49,999 INFO [Listener at localhost/42255] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46229 2023-02-14 03:53:50,008 WARN [Listener at localhost/34055] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:50,043 WARN [Listener at localhost/34055] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 03:53:50,046 WARN [Listener at localhost/34055] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:50,048 INFO [Listener at localhost/34055] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:50,052 INFO [Listener at localhost/34055] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/java.io.tmpdir/Jetty_localhost_43023_datanode____rk7rll/webapp 2023-02-14 03:53:50,149 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf0f156febc652196: Processing first storage report for DS-a4e18589-5411-42b9-becb-3384a440deb7 from datanode 5d365fbe-0c52-4882-bf12-29e6aef44dde 2023-02-14 03:53:50,149 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf0f156febc652196: from storage DS-a4e18589-5411-42b9-becb-3384a440deb7 node DatanodeRegistration(127.0.0.1:35271, datanodeUuid=5d365fbe-0c52-4882-bf12-29e6aef44dde, infoPort=41495, infoSecurePort=0, ipcPort=34055, storageInfo=lv=-57;cid=testClusterID;nsid=2104657353;c=1676346829676), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:50,149 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf0f156febc652196: Processing first storage report for DS-307d7c28-0803-4b8b-b7c0-1f6fe880d9e4 from datanode 5d365fbe-0c52-4882-bf12-29e6aef44dde 2023-02-14 03:53:50,149 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf0f156febc652196: from storage DS-307d7c28-0803-4b8b-b7c0-1f6fe880d9e4 node DatanodeRegistration(127.0.0.1:35271, datanodeUuid=5d365fbe-0c52-4882-bf12-29e6aef44dde, infoPort=41495, infoSecurePort=0, ipcPort=34055, storageInfo=lv=-57;cid=testClusterID;nsid=2104657353;c=1676346829676), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:50,196 INFO [Listener at localhost/34055] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43023 2023-02-14 03:53:50,217 WARN [Listener at localhost/41169] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:50,256 WARN [Listener at localhost/41169] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-14 03:53:50,262 WARN [Listener at localhost/41169] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-14 03:53:50,264 INFO [Listener at localhost/41169] log.Slf4jLog(67): jetty-6.1.26 2023-02-14 03:53:50,268 INFO [Listener at localhost/41169] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/java.io.tmpdir/Jetty_localhost_34103_datanode____.wn1412/webapp 2023-02-14 03:53:50,355 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5fe838ccede72c31: Processing first storage report for DS-8ed84e9a-5657-4856-9cb1-c51063fb72df from datanode 1e5929df-1979-4aff-b66f-e0e3c6547a97 2023-02-14 03:53:50,356 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5fe838ccede72c31: from storage DS-8ed84e9a-5657-4856-9cb1-c51063fb72df node DatanodeRegistration(127.0.0.1:37243, datanodeUuid=1e5929df-1979-4aff-b66f-e0e3c6547a97, infoPort=36823, infoSecurePort=0, ipcPort=41169, storageInfo=lv=-57;cid=testClusterID;nsid=2104657353;c=1676346829676), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:50,356 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5fe838ccede72c31: Processing first storage report for DS-2258ff8d-ed10-4984-b08f-b1fa81c94a5c from datanode 1e5929df-1979-4aff-b66f-e0e3c6547a97 2023-02-14 03:53:50,356 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5fe838ccede72c31: from storage DS-2258ff8d-ed10-4984-b08f-b1fa81c94a5c node DatanodeRegistration(127.0.0.1:37243, datanodeUuid=1e5929df-1979-4aff-b66f-e0e3c6547a97, infoPort=36823, infoSecurePort=0, ipcPort=41169, storageInfo=lv=-57;cid=testClusterID;nsid=2104657353;c=1676346829676), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:50,385 INFO [Listener at localhost/41169] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34103 2023-02-14 03:53:50,401 WARN [Listener at localhost/46385] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-14 03:53:50,520 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9a0e4d0ab321ccbd: Processing first storage report for DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f from datanode a43aa0f6-e723-4569-86af-29691e5aa956 2023-02-14 03:53:50,520 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9a0e4d0ab321ccbd: from storage DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f node DatanodeRegistration(127.0.0.1:43537, datanodeUuid=a43aa0f6-e723-4569-86af-29691e5aa956, infoPort=37583, infoSecurePort=0, ipcPort=46385, storageInfo=lv=-57;cid=testClusterID;nsid=2104657353;c=1676346829676), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:50,520 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9a0e4d0ab321ccbd: Processing first storage report for DS-280fede1-9270-489e-bed9-06b82101f287 from datanode a43aa0f6-e723-4569-86af-29691e5aa956 2023-02-14 03:53:50,520 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9a0e4d0ab321ccbd: from storage DS-280fede1-9270-489e-bed9-06b82101f287 node DatanodeRegistration(127.0.0.1:43537, datanodeUuid=a43aa0f6-e723-4569-86af-29691e5aa956, infoPort=37583, infoSecurePort=0, ipcPort=46385, storageInfo=lv=-57;cid=testClusterID;nsid=2104657353;c=1676346829676), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-14 03:53:50,614 DEBUG [Listener at localhost/46385] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed 2023-02-14 03:53:50,618 INFO [Listener at localhost/46385] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/zookeeper_0, clientPort=59369, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-14 03:53:50,619 INFO [Listener at localhost/46385] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59369 2023-02-14 03:53:50,620 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,621 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,648 INFO [Listener at localhost/46385] util.FSUtils(479): Created version file at hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 with version=8 2023-02-14 03:53:50,648 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/hbase-staging 2023-02-14 03:53:50,650 INFO [Listener at localhost/46385] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:50,651 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,651 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,651 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:50,651 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,651 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:50,651 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:50,652 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:41061 2023-02-14 03:53:50,652 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,653 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,655 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=master:41061 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:50,663 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:410610x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:50,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:41061-0x10050f78f070000 connected 2023-02-14 03:53:50,683 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:50,683 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:50,684 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:50,687 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41061 2023-02-14 03:53:50,687 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41061 2023-02-14 03:53:50,687 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41061 2023-02-14 03:53:50,689 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41061 2023-02-14 03:53:50,689 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41061 2023-02-14 03:53:50,690 INFO [Listener at localhost/46385] master.HMaster(439): hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99, hbase.cluster.distributed=false 2023-02-14 03:53:50,703 INFO [Listener at localhost/46385] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:50,703 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,703 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,703 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:50,703 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,704 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:50,704 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:50,704 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33791 2023-02-14 03:53:50,705 INFO [Listener at localhost/46385] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:50,706 DEBUG [Listener at localhost/46385] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:50,706 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,708 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,709 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33791 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:50,713 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:50,714 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:50,715 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:50,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33791-0x10050f78f070001 connected 2023-02-14 03:53:50,716 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:50,717 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33791 2023-02-14 03:53:50,717 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33791 2023-02-14 03:53:50,717 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33791 2023-02-14 03:53:50,722 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33791 2023-02-14 03:53:50,723 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33791 2023-02-14 03:53:50,739 INFO [Listener at localhost/46385] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:50,740 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,740 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,740 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:50,740 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,740 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:50,740 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:50,743 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33009 2023-02-14 03:53:50,744 INFO [Listener at localhost/46385] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:50,746 DEBUG [Listener at localhost/46385] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:50,747 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,748 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,750 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33009 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:50,754 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:330090x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:50,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33009-0x10050f78f070002 connected 2023-02-14 03:53:50,757 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:50,757 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:50,758 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:50,763 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33009 2023-02-14 03:53:50,763 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33009 2023-02-14 03:53:50,767 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33009 2023-02-14 03:53:50,770 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33009 2023-02-14 03:53:50,771 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33009 2023-02-14 03:53:50,787 INFO [Listener at localhost/46385] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:50,787 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,787 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,788 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:50,788 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:50,788 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:50,788 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:50,789 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33231 2023-02-14 03:53:50,789 INFO [Listener at localhost/46385] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:50,791 DEBUG [Listener at localhost/46385] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:50,791 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,793 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,794 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33231 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:50,802 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:332310x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:50,804 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:332310x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:50,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33231-0x10050f78f070003 connected 2023-02-14 03:53:50,805 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:53:50,805 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:50,811 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33231 2023-02-14 03:53:50,813 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33231 2023-02-14 03:53:50,814 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33231 2023-02-14 03:53:50,819 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33231 2023-02-14 03:53:50,819 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33231 2023-02-14 03:53:50,823 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:50,831 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:50,831 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:50,836 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:50,836 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:50,836 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:50,837 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:50,839 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:50,839 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:50,840 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,41061,1676346830650 from backup master directory 2023-02-14 03:53:50,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:50,844 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:50,844 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:50,844 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:50,844 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:50,870 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/hbase.id with ID: ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:50,887 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:50,890 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:50,916 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x5e60cde7 to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:50,923 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78ed3168, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:50,924 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:50,924 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-14 03:53:50,931 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:50,933 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store-tmp 2023-02-14 03:53:50,974 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:50,974 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 03:53:50,974 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:50,974 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:50,974 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 03:53:50,974 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:50,974 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:50,974 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676346830974Disabling compacts and flushes for region at 1676346830974Disabling writes for close at 1676346830974Writing region close event to WAL at 1676346830974Closed at 1676346830974 2023-02-14 03:53:50,975 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:50,980 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C41061%2C1676346830650, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/oldWALs, maxLogs=10 2023-02-14 03:53:51,004 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:51,006 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:51,007 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:51,011 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 2023-02-14 03:53:51,011 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK]] 2023-02-14 03:53:51,011 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:51,011 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:51,012 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:51,012 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:51,019 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:51,022 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-14 03:53:51,022 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-14 03:53:51,023 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:51,024 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:51,025 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:51,028 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:51,033 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:51,033 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66796154, jitterRate=-0.004659742116928101}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:51,033 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676346831012Initializing all the Stores at 1676346831013 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346831013Cleaning up temporary data from old regions at 1676346831025 (+12 ms)Cleaning up detritus from prior splits at 1676346831026 (+1 ms)Region opened successfully at 1676346831033 (+7 ms) 2023-02-14 03:53:51,034 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-14 03:53:51,035 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-14 03:53:51,035 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-14 03:53:51,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-14 03:53:51,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-14 03:53:51,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-14 03:53:51,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-14 03:53:51,037 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-14 03:53:51,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-14 03:53:51,054 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-14 03:53:51,054 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-14 03:53:51,055 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-14 03:53:51,055 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-14 03:53:51,055 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-14 03:53:51,057 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:51,058 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-14 03:53:51,058 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-14 03:53:51,059 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-14 03:53:51,060 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:51,060 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:51,060 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:51,060 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-14 03:53:51,060 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:51,061 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase9.apache.org,41061,1676346830650, sessionid=0x10050f78f070000, setting cluster-up flag (Was=false) 2023-02-14 03:53:51,067 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:51,071 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-14 03:53:51,072 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:51,076 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:51,084 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-14 03:53:51,084 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:51,085 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.hbase-snapshot/.tmp 2023-02-14 03:53:51,087 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:51,091 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,098 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676346861098 2023-02-14 03:53:51,098 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-14 03:53:51,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-14 03:53:51,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-14 03:53:51,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-14 03:53:51,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-14 03:53:51,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-14 03:53:51,101 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,103 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:51,103 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-14 03:53:51,103 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-14 03:53:51,103 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-14 03:53:51,103 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-14 03:53:51,105 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:51,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-14 03:53:51,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-14 03:53:51,115 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346831112,5,FailOnTimeoutGroup] 2023-02-14 03:53:51,119 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346831119,5,FailOnTimeoutGroup] 2023-02-14 03:53:51,120 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,121 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:51,123 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-14 03:53:51,123 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,123 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,124 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:51,126 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:51,126 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:51,128 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:51,132 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:51,141 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:51,141 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:51,141 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:51,141 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:51,142 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:51,142 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:51,154 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:51,156 DEBUG [RS:1;jenkins-hbase9:33009] zookeeper.ReadOnlyZKClient(139): Connect 0x79d6af4f to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:51,156 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:51,156 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:51,161 DEBUG [RS:2;jenkins-hbase9:33231] zookeeper.ReadOnlyZKClient(139): Connect 0x18c9e336 to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:51,161 DEBUG [RS:0;jenkins-hbase9:33791] zookeeper.ReadOnlyZKClient(139): Connect 0x67ad9aa4 to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:51,162 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:51,163 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:51,163 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:51,181 DEBUG [RS:1;jenkins-hbase9:33009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77114e05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:51,181 DEBUG [RS:1;jenkins-hbase9:33009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@362c443e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:51,186 DEBUG [RS:0;jenkins-hbase9:33791] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@696a7821, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:51,186 DEBUG [RS:0;jenkins-hbase9:33791] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d55f1d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:51,191 DEBUG [RS:2;jenkins-hbase9:33231] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@160e0e5f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:51,192 DEBUG [RS:2;jenkins-hbase9:33231] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12318846, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:51,199 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase9:33009 2023-02-14 03:53:51,199 INFO [RS:1;jenkins-hbase9:33009] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:51,199 INFO [RS:1;jenkins-hbase9:33009] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:51,199 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:51,202 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,41061,1676346830650 with isa=jenkins-hbase9.apache.org/172.31.2.10:33009, startcode=1676346830739 2023-02-14 03:53:51,202 DEBUG [RS:1;jenkins-hbase9:33009] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:51,204 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:33791 2023-02-14 03:53:51,204 INFO [RS:0;jenkins-hbase9:33791] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:51,204 INFO [RS:0;jenkins-hbase9:33791] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:51,204 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:51,205 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase9:33231 2023-02-14 03:53:51,205 INFO [RS:2;jenkins-hbase9:33231] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:51,205 INFO [RS:2;jenkins-hbase9:33231] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:51,205 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:51,205 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,41061,1676346830650 with isa=jenkins-hbase9.apache.org/172.31.2.10:33791, startcode=1676346830702 2023-02-14 03:53:51,205 DEBUG [RS:0;jenkins-hbase9:33791] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:51,205 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,41061,1676346830650 with isa=jenkins-hbase9.apache.org/172.31.2.10:33231, startcode=1676346830786 2023-02-14 03:53:51,206 DEBUG [RS:2;jenkins-hbase9:33231] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:51,207 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48167, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:51,208 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44113, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.7 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:51,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41061] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,209 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,212 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:51,212 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42255 2023-02-14 03:53:51,212 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:51,214 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:51,214 DEBUG [RS:2;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,214 WARN [RS:2;jenkins-hbase9:33231] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:51,214 INFO [RS:2;jenkins-hbase9:33231] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:51,214 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,237 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:51,237 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42255 2023-02-14 03:53:51,237 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:51,239 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36377, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:51,240 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,241 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:51,241 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42255 2023-02-14 03:53:51,241 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:51,245 DEBUG [RS:0;jenkins-hbase9:33791] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,245 DEBUG [RS:2;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,245 WARN [RS:0;jenkins-hbase9:33791] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:51,245 INFO [RS:0;jenkins-hbase9:33791] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:51,245 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,245 DEBUG [RS:2;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,246 DEBUG [RS:2;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,246 DEBUG [RS:1;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,246 WARN [RS:1;jenkins-hbase9:33009] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:51,246 INFO [RS:1;jenkins-hbase9:33009] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:51,247 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,247 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33231,1676346830786] 2023-02-14 03:53:51,247 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33009,1676346830739] 2023-02-14 03:53:51,248 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33791,1676346830702] 2023-02-14 03:53:51,248 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:51,249 INFO [RS:2;jenkins-hbase9:33231] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:51,252 INFO [RS:2;jenkins-hbase9:33231] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:51,253 INFO [RS:2;jenkins-hbase9:33231] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:51,253 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,253 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:51,255 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,255 DEBUG [RS:0;jenkins-hbase9:33791] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,255 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,255 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,255 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,255 DEBUG [RS:1;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,255 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,255 DEBUG [RS:0;jenkins-hbase9:33791] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,255 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,255 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:51,256 DEBUG [RS:1;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,256 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,256 DEBUG [RS:0;jenkins-hbase9:33791] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,256 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,256 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,256 DEBUG [RS:2;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,256 DEBUG [RS:1;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,257 DEBUG [RS:0;jenkins-hbase9:33791] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:51,257 INFO [RS:0;jenkins-hbase9:33791] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:51,257 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:51,258 INFO [RS:1;jenkins-hbase9:33009] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:51,260 INFO [RS:0;jenkins-hbase9:33791] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:51,261 INFO [RS:1;jenkins-hbase9:33009] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:51,261 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,261 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,261 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,263 INFO [RS:0;jenkins-hbase9:33791] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:51,263 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,267 INFO [RS:1;jenkins-hbase9:33009] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:51,267 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,267 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:51,268 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:51,269 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,269 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,270 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,270 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:51,271 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:51,271 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:0;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,271 DEBUG [RS:1;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:51,278 INFO [RS:2;jenkins-hbase9:33231] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:51,278 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33231,1676346830786-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,283 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,283 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,284 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,285 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,285 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,285 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,295 INFO [RS:2;jenkins-hbase9:33231] regionserver.Replication(203): jenkins-hbase9.apache.org,33231,1676346830786 started 2023-02-14 03:53:51,295 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,33231,1676346830786, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33231, sessionid=0x10050f78f070003 2023-02-14 03:53:51,295 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:51,295 DEBUG [RS:2;jenkins-hbase9:33231] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,295 DEBUG [RS:2;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33231,1676346830786' 2023-02-14 03:53:51,295 DEBUG [RS:2;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:51,295 DEBUG [RS:2;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:51,296 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:51,296 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:51,296 DEBUG [RS:2;jenkins-hbase9:33231] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:51,296 DEBUG [RS:2;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33231,1676346830786' 2023-02-14 03:53:51,296 DEBUG [RS:2;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:51,296 DEBUG [RS:2;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:51,297 DEBUG [RS:2;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:51,297 INFO [RS:2;jenkins-hbase9:33231] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:51,297 INFO [RS:2;jenkins-hbase9:33231] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:51,298 INFO [RS:0;jenkins-hbase9:33791] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:51,298 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33791,1676346830702-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,302 INFO [RS:1;jenkins-hbase9:33009] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:51,302 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33009,1676346830739-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:51,312 INFO [RS:0;jenkins-hbase9:33791] regionserver.Replication(203): jenkins-hbase9.apache.org,33791,1676346830702 started 2023-02-14 03:53:51,312 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,33791,1676346830702, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33791, sessionid=0x10050f78f070001 2023-02-14 03:53:51,312 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:51,312 DEBUG [RS:0;jenkins-hbase9:33791] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,312 DEBUG [RS:0;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33791,1676346830702' 2023-02-14 03:53:51,312 DEBUG [RS:0;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33791,1676346830702' 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:51,313 DEBUG [RS:0;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:51,314 DEBUG [RS:0;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:51,314 INFO [RS:0;jenkins-hbase9:33791] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:51,314 INFO [RS:0;jenkins-hbase9:33791] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:51,317 INFO [RS:1;jenkins-hbase9:33009] regionserver.Replication(203): jenkins-hbase9.apache.org,33009,1676346830739 started 2023-02-14 03:53:51,317 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,33009,1676346830739, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33009, sessionid=0x10050f78f070002 2023-02-14 03:53:51,317 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:51,317 DEBUG [RS:1;jenkins-hbase9:33009] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,317 DEBUG [RS:1;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33009,1676346830739' 2023-02-14 03:53:51,317 DEBUG [RS:1;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:51,318 DEBUG [RS:1;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:51,318 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:51,318 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:51,318 DEBUG [RS:1;jenkins-hbase9:33009] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,318 DEBUG [RS:1;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33009,1676346830739' 2023-02-14 03:53:51,318 DEBUG [RS:1;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:51,319 DEBUG [RS:1;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:51,319 DEBUG [RS:1;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:51,319 INFO [RS:1;jenkins-hbase9:33009] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:51,319 INFO [RS:1;jenkins-hbase9:33009] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:51,400 INFO [RS:2;jenkins-hbase9:33231] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33231%2C1676346830786, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:51,417 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:51,417 INFO [RS:0;jenkins-hbase9:33791] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33791%2C1676346830702, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:51,419 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:51,419 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:51,422 INFO [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33009%2C1676346830739, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:51,426 INFO [RS:2;jenkins-hbase9:33231] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786/jenkins-hbase9.apache.org%2C33231%2C1676346830786.1676346831401 2023-02-14 03:53:51,427 DEBUG [RS:2;jenkins-hbase9:33231] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK]] 2023-02-14 03:53:51,440 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:51,440 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:51,441 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:51,449 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:51,449 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:51,450 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:51,450 INFO [RS:0;jenkins-hbase9:33791] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702/jenkins-hbase9.apache.org%2C33791%2C1676346830702.1676346831418 2023-02-14 03:53:51,450 DEBUG [RS:0;jenkins-hbase9:33791] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK]] 2023-02-14 03:53:51,455 INFO [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739/jenkins-hbase9.apache.org%2C33009%2C1676346830739.1676346831422 2023-02-14 03:53:51,459 DEBUG [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK], DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK]] 2023-02-14 03:53:51,642 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:51,644 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 03:53:51,646 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info 2023-02-14 03:53:51,648 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 03:53:51,650 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:51,650 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 03:53:51,654 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:51,654 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 03:53:51,655 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:51,655 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 03:53:51,657 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table 2023-02-14 03:53:51,657 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 03:53:51,658 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:51,659 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740 2023-02-14 03:53:51,660 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740 2023-02-14 03:53:51,664 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 03:53:51,671 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 03:53:51,674 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:51,675 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71243081, jitterRate=0.061604633927345276}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 03:53:51,675 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676346831642Initializing all the Stores at 1676346831643 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346831643Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346831643Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346831643Cleaning up temporary data from old regions at 1676346831662 (+19 ms)Cleaning up detritus from prior splits at 1676346831662Region opened successfully at 1676346831675 (+13 ms) 2023-02-14 03:53:51,675 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 03:53:51,675 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 03:53:51,675 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 03:53:51,675 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 03:53:51,675 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 03:53:51,676 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 03:53:51,676 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676346831675Disabling compacts and flushes for region at 1676346831675Disabling writes for close at 1676346831675Writing region close event to WAL at 1676346831676 (+1 ms)Closed at 1676346831676 2023-02-14 03:53:51,678 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:51,678 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-14 03:53:51,678 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-14 03:53:51,681 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-14 03:53:51,683 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-14 03:53:51,833 DEBUG [jenkins-hbase9:41061] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-14 03:53:51,833 DEBUG [jenkins-hbase9:41061] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:51,836 DEBUG [jenkins-hbase9:41061] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:51,836 DEBUG [jenkins-hbase9:41061] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:51,836 DEBUG [jenkins-hbase9:41061] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:51,836 DEBUG [jenkins-hbase9:41061] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:51,837 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33009,1676346830739, state=OPENING 2023-02-14 03:53:51,838 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-14 03:53:51,839 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:51,840 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 03:53:51,840 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33009,1676346830739}] 2023-02-14 03:53:51,995 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:51,995 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:51,997 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52998, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 03:53:52,002 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-14 03:53:52,002 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:52,004 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33009%2C1676346830739.meta, suffix=.meta, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:52,023 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:52,023 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:52,024 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:52,033 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739/jenkins-hbase9.apache.org%2C33009%2C1676346830739.meta.1676346832005.meta 2023-02-14 03:53:52,033 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK]] 2023-02-14 03:53:52,033 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:52,034 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-14 03:53:52,034 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-14 03:53:52,035 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-14 03:53:52,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-14 03:53:52,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:52,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-14 03:53:52,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-14 03:53:52,038 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 03:53:52,039 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info 2023-02-14 03:53:52,039 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info 2023-02-14 03:53:52,040 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 03:53:52,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:52,041 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 03:53:52,042 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:52,042 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:52,043 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 03:53:52,043 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:52,044 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 03:53:52,047 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table 2023-02-14 03:53:52,047 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table 2023-02-14 03:53:52,048 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 03:53:52,049 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:52,050 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740 2023-02-14 03:53:52,051 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740 2023-02-14 03:53:52,054 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 03:53:52,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 03:53:52,057 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72772499, jitterRate=0.08439473807811737}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 03:53:52,057 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676346832036Writing region info on filesystem at 1676346832036Initializing all the Stores at 1676346832037 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346832037Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346832038 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346832038Cleaning up temporary data from old regions at 1676346832053 (+15 ms)Cleaning up detritus from prior splits at 1676346832053Running coprocessor post-open hooks at 1676346832057 (+4 ms)Region opened successfully at 1676346832057 2023-02-14 03:53:52,059 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676346831995 2023-02-14 03:53:52,069 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-14 03:53:52,076 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-14 03:53:52,079 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33009,1676346830739, state=OPEN 2023-02-14 03:53:52,084 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-14 03:53:52,084 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 03:53:52,088 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-14 03:53:52,088 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33009,1676346830739 in 244 msec 2023-02-14 03:53:52,092 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-14 03:53:52,093 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 410 msec 2023-02-14 03:53:52,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0080 sec 2023-02-14 03:53:52,097 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676346830831, completionTime=-1 2023-02-14 03:53:52,097 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-14 03:53:52,097 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-14 03:53:52,100 DEBUG [hconnection-0x623172f2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:52,102 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53006, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:52,104 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-14 03:53:52,104 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676346892104 2023-02-14 03:53:52,104 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676346952104 2023-02-14 03:53:52,104 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 6 msec 2023-02-14 03:53:52,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41061,1676346830650-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:52,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41061,1676346830650-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:52,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41061,1676346830650-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:52,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:41061, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:52,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:52,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-14 03:53:52,115 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:52,116 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-14 03:53:52,116 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-14 03:53:52,118 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:52,119 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:52,151 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/hbase/namespace/ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,152 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/hbase/namespace/ee39c662a4152a296031f22713f8127e empty. 2023-02-14 03:53:52,152 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/hbase/namespace/ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,152 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-14 03:53:52,182 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:52,184 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => ee39c662a4152a296031f22713f8127e, NAME => 'hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp 2023-02-14 03:53:52,204 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:52,204 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing ee39c662a4152a296031f22713f8127e, disabling compactions & flushes 2023-02-14 03:53:52,204 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,204 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,204 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. after waiting 0 ms 2023-02-14 03:53:52,204 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,204 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,204 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for ee39c662a4152a296031f22713f8127e: Waiting for close lock at 1676346832204Disabling compacts and flushes for region at 1676346832204Disabling writes for close at 1676346832204Writing region close event to WAL at 1676346832204Closed at 1676346832204 2023-02-14 03:53:52,207 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:52,208 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346832208"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346832208"}]},"ts":"1676346832208"} 2023-02-14 03:53:52,211 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:52,212 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:52,212 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346832212"}]},"ts":"1676346832212"} 2023-02-14 03:53:52,214 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-14 03:53:52,219 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:52,220 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:52,220 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:52,220 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:52,220 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:52,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN}] 2023-02-14 03:53:52,223 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN 2023-02-14 03:53:52,224 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33231,1676346830786; forceNewPlan=false, retain=false 2023-02-14 03:53:52,374 INFO [jenkins-hbase9:41061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:52,376 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ee39c662a4152a296031f22713f8127e, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:52,376 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346832375"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346832375"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346832375"}]},"ts":"1676346832375"} 2023-02-14 03:53:52,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure ee39c662a4152a296031f22713f8127e, server=jenkins-hbase9.apache.org,33231,1676346830786}] 2023-02-14 03:53:52,537 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:52,537 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:52,539 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-14 03:53:52,544 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,544 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee39c662a4152a296031f22713f8127e, NAME => 'hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:52,544 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,544 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:52,544 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,544 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,546 INFO [StoreOpener-ee39c662a4152a296031f22713f8127e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,548 DEBUG [StoreOpener-ee39c662a4152a296031f22713f8127e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info 2023-02-14 03:53:52,548 DEBUG [StoreOpener-ee39c662a4152a296031f22713f8127e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info 2023-02-14 03:53:52,548 INFO [StoreOpener-ee39c662a4152a296031f22713f8127e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee39c662a4152a296031f22713f8127e columnFamilyName info 2023-02-14 03:53:52,549 INFO [StoreOpener-ee39c662a4152a296031f22713f8127e-1] regionserver.HStore(310): Store=ee39c662a4152a296031f22713f8127e/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:52,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,553 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:52,558 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:52,559 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened ee39c662a4152a296031f22713f8127e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63183312, jitterRate=-0.058495283126831055}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:52,559 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for ee39c662a4152a296031f22713f8127e: Running coprocessor pre-open hook at 1676346832545Writing region info on filesystem at 1676346832545Initializing all the Stores at 1676346832545Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346832546 (+1 ms)Cleaning up temporary data from old regions at 1676346832551 (+5 ms)Cleaning up detritus from prior splits at 1676346832551Running coprocessor post-open hooks at 1676346832559 (+8 ms)Region opened successfully at 1676346832559 2023-02-14 03:53:52,561 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e., pid=6, masterSystemTime=1676346832537 2023-02-14 03:53:52,566 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,567 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:52,567 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ee39c662a4152a296031f22713f8127e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:52,567 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346832567"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346832567"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346832567"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346832567"}]},"ts":"1676346832567"} 2023-02-14 03:53:52,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-14 03:53:52,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure ee39c662a4152a296031f22713f8127e, server=jenkins-hbase9.apache.org,33231,1676346830786 in 191 msec 2023-02-14 03:53:52,577 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-14 03:53:52,577 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN in 352 msec 2023-02-14 03:53:52,578 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:52,579 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346832579"}]},"ts":"1676346832579"} 2023-02-14 03:53:52,581 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-14 03:53:52,583 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:52,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 469 msec 2023-02-14 03:53:52,618 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-14 03:53:52,620 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-14 03:53:52,621 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:52,624 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:52,625 INFO [RS-EventLoopGroup-12-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54510, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:52,627 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-14 03:53:52,644 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-14 03:53:52,648 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 20 msec 2023-02-14 03:53:52,661 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-14 03:53:52,672 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-14 03:53:52,684 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 20 msec 2023-02-14 03:53:52,696 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-14 03:53:52,702 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-14 03:53:52,702 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.858sec 2023-02-14 03:53:52,707 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-14 03:53:52,707 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-14 03:53:52,707 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-14 03:53:52,707 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41061,1676346830650-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-14 03:53:52,707 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,41061,1676346830650-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-14 03:53:52,709 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-14 03:53:52,740 DEBUG [Listener at localhost/46385] zookeeper.ReadOnlyZKClient(139): Connect 0x4b95821d to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:52,745 DEBUG [Listener at localhost/46385] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c85628c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:52,747 DEBUG [hconnection-0x5d9d771d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:52,751 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53022, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:52,753 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:52,756 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-14 03:53:52,756 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:52,757 INFO [Listener at localhost/46385] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-14 03:53:52,757 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-14 03:53:52,759 DEBUG [Listener at localhost/46385] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-14 03:53:52,761 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:55826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-14 03:53:52,762 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.HMaster$4(2102): Client=jenkins//172.31.2.10 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-14 03:53:52,766 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:52,766 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-14 03:53:52,767 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:52,769 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:52,770 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec empty. 2023-02-14 03:53:52,770 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:52,770 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-14 03:53:52,785 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:52,786 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => a93dd5cbcf0c0282a4de644346a8b9ec, NAME => 'restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp 2023-02-14 03:53:52,797 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:52,798 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing a93dd5cbcf0c0282a4de644346a8b9ec, disabling compactions & flushes 2023-02-14 03:53:52,798 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:52,798 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:52,798 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. after waiting 0 ms 2023-02-14 03:53:52,798 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:52,798 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:52,798 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for a93dd5cbcf0c0282a4de644346a8b9ec: Waiting for close lock at 1676346832798Disabling compacts and flushes for region at 1676346832798Disabling writes for close at 1676346832798Writing region close event to WAL at 1676346832798Closed at 1676346832798 2023-02-14 03:53:52,801 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:52,802 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346832801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346832801"}]},"ts":"1676346832801"} 2023-02-14 03:53:52,803 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:52,804 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:52,804 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346832804"}]},"ts":"1676346832804"} 2023-02-14 03:53:52,806 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-14 03:53:52,810 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:52,810 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:52,810 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:52,810 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:52,810 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:52,811 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN}] 2023-02-14 03:53:52,812 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN 2023-02-14 03:53:52,813 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=false 2023-02-14 03:53:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:52,964 INFO [jenkins-hbase9:41061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:52,965 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a93dd5cbcf0c0282a4de644346a8b9ec, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:52,965 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346832965"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346832965"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346832965"}]},"ts":"1676346832965"} 2023-02-14 03:53:52,968 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure a93dd5cbcf0c0282a4de644346a8b9ec, server=jenkins-hbase9.apache.org,33009,1676346830739}] 2023-02-14 03:53:53,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:53,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:53,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a93dd5cbcf0c0282a4de644346a8b9ec, NAME => 'restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:53,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:53,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,127 INFO [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,129 DEBUG [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/family 2023-02-14 03:53:53,129 DEBUG [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/family 2023-02-14 03:53:53,129 INFO [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a93dd5cbcf0c0282a4de644346a8b9ec columnFamilyName family 2023-02-14 03:53:53,130 INFO [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] regionserver.HStore(310): Store=a93dd5cbcf0c0282a4de644346a8b9ec/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:53,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:53,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:53,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened a93dd5cbcf0c0282a4de644346a8b9ec; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59801684, jitterRate=-0.10888546705245972}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:53,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for a93dd5cbcf0c0282a4de644346a8b9ec: Running coprocessor pre-open hook at 1676346833126Writing region info on filesystem at 1676346833126Initializing all the Stores at 1676346833127 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346833127Cleaning up temporary data from old regions at 1676346833132 (+5 ms)Cleaning up detritus from prior splits at 1676346833132Running coprocessor post-open hooks at 1676346833138 (+6 ms)Region opened successfully at 1676346833138 2023-02-14 03:53:53,139 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec., pid=11, masterSystemTime=1676346833120 2023-02-14 03:53:53,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:53,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:53,141 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a93dd5cbcf0c0282a4de644346a8b9ec, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:53,142 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346833141"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346833141"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346833141"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346833141"}]},"ts":"1676346833141"} 2023-02-14 03:53:53,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-14 03:53:53,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure a93dd5cbcf0c0282a4de644346a8b9ec, server=jenkins-hbase9.apache.org,33009,1676346830739 in 176 msec 2023-02-14 03:53:53,151 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-14 03:53:53,151 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN in 336 msec 2023-02-14 03:53:53,152 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:53,152 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346833152"}]},"ts":"1676346833152"} 2023-02-14 03:53:53,154 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-14 03:53:53,157 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:53,158 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 394 msec 2023-02-14 03:53:53,326 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-14 03:53:53,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-14 03:53:53,372 INFO [Listener at localhost/46385] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-14 03:53:53,373 DEBUG [Listener at localhost/46385] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-14 03:53:53,373 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:53,376 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-14 03:53:53,376 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:53,377 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-14 03:53:53,381 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.HMaster$4(2102): Client=jenkins//172.31.2.10 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:53,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-14 03:53:53,384 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:53,384 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-14 03:53:53,385 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:53,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:53,388 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,389 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 empty. 2023-02-14 03:53:53,389 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,389 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-14 03:53:53,412 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:53,414 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 1da7aa25683c42faa92bcbdf37a68280, NAME => 'restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp 2023-02-14 03:53:53,430 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:53,430 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 1da7aa25683c42faa92bcbdf37a68280, disabling compactions & flushes 2023-02-14 03:53:53,430 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,430 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,430 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. after waiting 0 ms 2023-02-14 03:53:53,430 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,430 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,430 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 1da7aa25683c42faa92bcbdf37a68280: Waiting for close lock at 1676346833430Disabling compacts and flushes for region at 1676346833430Disabling writes for close at 1676346833430Writing region close event to WAL at 1676346833430Closed at 1676346833430 2023-02-14 03:53:53,433 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:53,434 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346833433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346833433"}]},"ts":"1676346833433"} 2023-02-14 03:53:53,435 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:53,437 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:53,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346833437"}]},"ts":"1676346833437"} 2023-02-14 03:53:53,438 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-14 03:53:53,442 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:53,443 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:53,443 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:53,443 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:53,443 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:53,443 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN}] 2023-02-14 03:53:53,445 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN 2023-02-14 03:53:53,446 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=false 2023-02-14 03:53:53,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:53,596 INFO [jenkins-hbase9:41061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:53,597 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1da7aa25683c42faa92bcbdf37a68280, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:53,597 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346833597"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346833597"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346833597"}]},"ts":"1676346833597"} 2023-02-14 03:53:53,599 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 1da7aa25683c42faa92bcbdf37a68280, server=jenkins-hbase9.apache.org,33009,1676346830739}] 2023-02-14 03:53:53,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:53,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1da7aa25683c42faa92bcbdf37a68280, NAME => 'restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:53,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:53,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,758 INFO [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,759 DEBUG [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/family 2023-02-14 03:53:53,759 DEBUG [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/family 2023-02-14 03:53:53,760 INFO [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1da7aa25683c42faa92bcbdf37a68280 columnFamilyName family 2023-02-14 03:53:53,760 INFO [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] regionserver.HStore(310): Store=1da7aa25683c42faa92bcbdf37a68280/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:53,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:53,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:53,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 1da7aa25683c42faa92bcbdf37a68280; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59066384, jitterRate=-0.1198422908782959}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:53,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 1da7aa25683c42faa92bcbdf37a68280: Running coprocessor pre-open hook at 1676346833756Writing region info on filesystem at 1676346833756Initializing all the Stores at 1676346833757 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346833757Cleaning up temporary data from old regions at 1676346833762 (+5 ms)Cleaning up detritus from prior splits at 1676346833763 (+1 ms)Running coprocessor post-open hooks at 1676346833768 (+5 ms)Region opened successfully at 1676346833768 2023-02-14 03:53:53,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280., pid=14, masterSystemTime=1676346833751 2023-02-14 03:53:53,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:53,771 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1da7aa25683c42faa92bcbdf37a68280, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:53,772 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346833771"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346833771"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346833771"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346833771"}]},"ts":"1676346833771"} 2023-02-14 03:53:53,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-14 03:53:53,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 1da7aa25683c42faa92bcbdf37a68280, server=jenkins-hbase9.apache.org,33009,1676346830739 in 174 msec 2023-02-14 03:53:53,778 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-14 03:53:53,778 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN in 332 msec 2023-02-14 03:53:53,778 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:53,779 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346833778"}]},"ts":"1676346833778"} 2023-02-14 03:53:53,780 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-14 03:53:53,783 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:53,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 402 msec 2023-02-14 03:53:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-14 03:53:53,990 INFO [Listener at localhost/46385] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-14 03:53:53,990 DEBUG [Listener at localhost/46385] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-14 03:53:53,991 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:53,993 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-14 03:53:53,994 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:53,994 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-14 03:53:53,995 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.HMaster$4(2102): Client=jenkins//172.31.2.10 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:53,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-14 03:53:54,000 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-14 03:53:54,000 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-14 03:53:54,001 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-14 03:53:54,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:54,003 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,003 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d empty. 2023-02-14 03:53:54,004 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,004 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-14 03:53:54,019 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-14 03:53:54,021 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => bd6d9dbbd3807b958cb89c661120405d, NAME => 'restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.tmp 2023-02-14 03:53:54,032 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:54,032 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing bd6d9dbbd3807b958cb89c661120405d, disabling compactions & flushes 2023-02-14 03:53:54,032 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,032 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,032 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. after waiting 0 ms 2023-02-14 03:53:54,032 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,033 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,033 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for bd6d9dbbd3807b958cb89c661120405d: Waiting for close lock at 1676346834032Disabling compacts and flushes for region at 1676346834032Disabling writes for close at 1676346834032Writing region close event to WAL at 1676346834033 (+1 ms)Closed at 1676346834033 2023-02-14 03:53:54,035 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-14 03:53:54,036 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346834036"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676346834036"}]},"ts":"1676346834036"} 2023-02-14 03:53:54,038 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-14 03:53:54,039 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-14 03:53:54,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346834039"}]},"ts":"1676346834039"} 2023-02-14 03:53:54,041 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-14 03:53:54,044 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-14 03:53:54,045 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-14 03:53:54,045 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-14 03:53:54,045 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-14 03:53:54,045 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-14 03:53:54,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN}] 2023-02-14 03:53:54,047 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN 2023-02-14 03:53:54,048 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=false 2023-02-14 03:53:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:54,199 INFO [jenkins-hbase9:41061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:54,200 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=bd6d9dbbd3807b958cb89c661120405d, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:54,200 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346834200"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346834200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346834200"}]},"ts":"1676346834200"} 2023-02-14 03:53:54,202 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure bd6d9dbbd3807b958cb89c661120405d, server=jenkins-hbase9.apache.org,33009,1676346830739}] 2023-02-14 03:53:54,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:54,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd6d9dbbd3807b958cb89c661120405d, NAME => 'restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:54,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:54,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,379 INFO [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,381 DEBUG [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/family 2023-02-14 03:53:54,381 DEBUG [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/family 2023-02-14 03:53:54,382 INFO [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd6d9dbbd3807b958cb89c661120405d columnFamilyName family 2023-02-14 03:53:54,383 INFO [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] regionserver.HStore(310): Store=bd6d9dbbd3807b958cb89c661120405d/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:54,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-14 03:53:54,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened bd6d9dbbd3807b958cb89c661120405d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63620671, jitterRate=-0.05197812616825104}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:54,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for bd6d9dbbd3807b958cb89c661120405d: Running coprocessor pre-open hook at 1676346834373Writing region info on filesystem at 1676346834373Initializing all the Stores at 1676346834374 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346834374Cleaning up temporary data from old regions at 1676346834385 (+11 ms)Cleaning up detritus from prior splits at 1676346834386 (+1 ms)Running coprocessor post-open hooks at 1676346834397 (+11 ms)Region opened successfully at 1676346834397 2023-02-14 03:53:54,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d., pid=17, masterSystemTime=1676346834354 2023-02-14 03:53:54,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,401 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=bd6d9dbbd3807b958cb89c661120405d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:54,401 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346834401"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346834401"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346834401"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346834401"}]},"ts":"1676346834401"} 2023-02-14 03:53:54,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-14 03:53:54,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure bd6d9dbbd3807b958cb89c661120405d, server=jenkins-hbase9.apache.org,33009,1676346830739 in 201 msec 2023-02-14 03:53:54,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-14 03:53:54,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN in 360 msec 2023-02-14 03:53:54,408 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-14 03:53:54,409 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676346834408"}]},"ts":"1676346834408"} 2023-02-14 03:53:54,410 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-14 03:53:54,413 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-14 03:53:54,416 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 418 msec 2023-02-14 03:53:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-14 03:53:54,605 INFO [Listener at localhost/46385] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-14 03:53:54,605 DEBUG [Listener at localhost/46385] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-14 03:53:54,606 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:54,608 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-14 03:53:54,609 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:54,609 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-14 03:53:54,609 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:54,612 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:54,615 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:54,618 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:54,618 INFO [Listener at localhost/46385] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-14 03:53:54,620 INFO [Listener at localhost/46385] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-14 03:53:54,620 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(88): Shutting down HBase cluster 2023-02-14 03:53:54,620 INFO [Listener at localhost/46385] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase9:41061,5,FailOnTimeoutGroup] 2023-02-14 03:53:54,620 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,41061,1676346830650' ***** 2023-02-14 03:53:54,620 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-14 03:53:54,621 DEBUG [M:0;jenkins-hbase9:41061] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23848f5b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:54,622 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,41061,1676346830650 2023-02-14 03:53:54,622 DEBUG [M:0;jenkins-hbase9:41061] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e60cde7 to 127.0.0.1:59369 2023-02-14 03:53:54,622 DEBUG [M:0;jenkins-hbase9:41061] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,622 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,41061,1676346830650; all regions closed. 2023-02-14 03:53:54,622 DEBUG [M:0;jenkins-hbase9:41061] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,622 DEBUG [M:0;jenkins-hbase9:41061] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-14 03:53:54,622 DEBUG [M:0;jenkins-hbase9:41061] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-14 03:53:54,622 INFO [M:0;jenkins-hbase9:41061] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-14 03:53:54,622 INFO [M:0;jenkins-hbase9:41061] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-14 03:53:54,623 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-14 03:53:54,623 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346831112] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346831112,5,FailOnTimeoutGroup] 2023-02-14 03:53:54,623 INFO [M:0;jenkins-hbase9:41061] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-02-14 03:53:54,623 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346831119] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346831119,5,FailOnTimeoutGroup] 2023-02-14 03:53:54,623 DEBUG [M:0;jenkins-hbase9:41061] master.HMaster(1502): Stopping service threads 2023-02-14 03:53:54,624 INFO [M:0;jenkins-hbase9:41061] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-14 03:53:54,625 INFO [M:0;jenkins-hbase9:41061] procedure2.ProcedureExecutor(629): Stopping 2023-02-14 03:53:54,625 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:54,625 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:54,625 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:54,625 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:54,625 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:54,625 DEBUG [Listener at localhost/46385] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-14 03:53:54,626 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,33791,1676346830702' ***** 2023-02-14 03:53:54,626 ERROR [M:0;jenkins-hbase9:41061] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-14 03:53:54,626 INFO [M:0;jenkins-hbase9:41061] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-14 03:53:54,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:54,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:54,626 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:53:54,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:54,626 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-14 03:53:54,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:54,627 INFO [RS:0;jenkins-hbase9:33791] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:54,627 DEBUG [M:0;jenkins-hbase9:41061] zookeeper.ZKUtil(398): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-14 03:53:54,627 INFO [RS:0;jenkins-hbase9:33791] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:54,627 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:54,627 INFO [RS:0;jenkins-hbase9:33791] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:54,627 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,33009,1676346830739' ***** 2023-02-14 03:53:54,628 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:54,627 WARN [M:0;jenkins-hbase9:41061] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-14 03:53:54,628 DEBUG [RS:0;jenkins-hbase9:33791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67ad9aa4 to 127.0.0.1:59369 2023-02-14 03:53:54,628 INFO [M:0;jenkins-hbase9:41061] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-14 03:53:54,628 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:53:54,628 DEBUG [RS:0;jenkins-hbase9:33791] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,629 INFO [RS:1;jenkins-hbase9:33009] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:54,629 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,33791,1676346830702; all regions closed. 2023-02-14 03:53:54,629 INFO [RS:1;jenkins-hbase9:33009] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:54,629 INFO [M:0;jenkins-hbase9:41061] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-14 03:53:54,629 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,33231,1676346830786' ***** 2023-02-14 03:53:54,629 INFO [RS:1;jenkins-hbase9:33009] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:54,629 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:54,630 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:54,630 DEBUG [M:0;jenkins-hbase9:41061] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 03:53:54,631 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:54,631 DEBUG [M:0;jenkins-hbase9:41061] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:54,631 DEBUG [M:0;jenkins-hbase9:41061] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 03:53:54,631 DEBUG [M:0;jenkins-hbase9:41061] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:54,630 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:53:54,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing a93dd5cbcf0c0282a4de644346a8b9ec, disabling compactions & flushes 2023-02-14 03:53:54,631 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,632 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.18 KB heapSize=76.14 KB 2023-02-14 03:53:54,632 INFO [RS:2;jenkins-hbase9:33231] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:53:54,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:54,632 INFO [RS:2;jenkins-hbase9:33231] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:53:54,632 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:54,632 INFO [RS:2;jenkins-hbase9:33231] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:53:54,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:54,632 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:53:54,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. after waiting 0 ms 2023-02-14 03:53:54,632 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(3304): Received CLOSE for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:54,632 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:54,634 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:54,634 DEBUG [RS:1;jenkins-hbase9:33009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79d6af4f to 127.0.0.1:59369 2023-02-14 03:53:54,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:54,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing ee39c662a4152a296031f22713f8127e, disabling compactions & flushes 2023-02-14 03:53:54,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:54,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:54,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. after waiting 0 ms 2023-02-14 03:53:54,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:54,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing ee39c662a4152a296031f22713f8127e 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-14 03:53:54,635 DEBUG [RS:1;jenkins-hbase9:33009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,636 INFO [RS:1;jenkins-hbase9:33009] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:54,636 INFO [RS:1;jenkins-hbase9:33009] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:54,636 INFO [RS:1;jenkins-hbase9:33009] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:54,636 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-14 03:53:54,634 DEBUG [RS:2;jenkins-hbase9:33231] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18c9e336 to 127.0.0.1:59369 2023-02-14 03:53:54,636 DEBUG [RS:2;jenkins-hbase9:33231] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,636 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-14 03:53:54,636 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1479): Online Regions={ee39c662a4152a296031f22713f8127e=hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.} 2023-02-14 03:53:54,636 DEBUG [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1505): Waiting on ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:54,639 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1475): Waiting on 4 regions to close 2023-02-14 03:53:54,639 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1479): Online Regions={a93dd5cbcf0c0282a4de644346a8b9ec=restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec., 1588230740=hbase:meta,,1.1588230740, bd6d9dbbd3807b958cb89c661120405d=restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d., 1da7aa25683c42faa92bcbdf37a68280=restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.} 2023-02-14 03:53:54,639 DEBUG [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1505): Waiting on 1588230740, 1da7aa25683c42faa92bcbdf37a68280, a93dd5cbcf0c0282a4de644346a8b9ec, bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:54,640 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 03:53:54,640 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 03:53:54,640 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 03:53:54,640 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 03:53:54,640 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 03:53:54,641 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-14 03:53:54,645 DEBUG [RS:0;jenkins-hbase9:33791] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:53:54,645 INFO [RS:0;jenkins-hbase9:33791] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33791%2C1676346830702:(num 1676346831418) 2023-02-14 03:53:54,645 DEBUG [RS:0;jenkins-hbase9:33791] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,645 INFO [RS:0;jenkins-hbase9:33791] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:54,646 INFO [RS:0;jenkins-hbase9:33791] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:54,646 INFO [RS:0;jenkins-hbase9:33791] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:54,646 INFO [RS:0;jenkins-hbase9:33791] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:54,646 INFO [RS:0;jenkins-hbase9:33791] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:54,647 INFO [RS:0;jenkins-hbase9:33791] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33791 2023-02-14 03:53:54,648 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:54,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 03:53:54,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:54,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for a93dd5cbcf0c0282a4de644346a8b9ec: Waiting for close lock at 1676346834631Running coprocessor pre-close hooks at 1676346834631Disabling compacts and flushes for region at 1676346834631Disabling writes for close at 1676346834634 (+3 ms)Writing region close event to WAL at 1676346834640 (+6 ms)Running coprocessor post-close hooks at 1676346834655 (+15 ms)Closed at 1676346834655 2023-02-14 03:53:54,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:54,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing bd6d9dbbd3807b958cb89c661120405d, disabling compactions & flushes 2023-02-14 03:53:54,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. after waiting 0 ms 2023-02-14 03:53:54,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,666 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:54,674 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 03:53:54,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for bd6d9dbbd3807b958cb89c661120405d: Waiting for close lock at 1676346834656Running coprocessor pre-close hooks at 1676346834656Disabling compacts and flushes for region at 1676346834656Disabling writes for close at 1676346834656Writing region close event to WAL at 1676346834660 (+4 ms)Running coprocessor post-close hooks at 1676346834675 (+15 ms)Closed at 1676346834675 2023-02-14 03:53:54,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:54,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 1da7aa25683c42faa92bcbdf37a68280, disabling compactions & flushes 2023-02-14 03:53:54,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:54,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:54,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. after waiting 0 ms 2023-02-14 03:53:54,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:54,685 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/.tmp/info/0664f2c4f1e94c53aa17bec03a18be22 2023-02-14 03:53:54,685 INFO [M:0;jenkins-hbase9:41061] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.18 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2445e6e37b574aa087d5428803903794 2023-02-14 03:53:54,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/.tmp/info/d2d886a8bd7e4d22bd29330c4a2bbca9 2023-02-14 03:53:54,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-14 03:53:54,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:54,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 1da7aa25683c42faa92bcbdf37a68280: Waiting for close lock at 1676346834676Running coprocessor pre-close hooks at 1676346834676Disabling compacts and flushes for region at 1676346834677 (+1 ms)Disabling writes for close at 1676346834679 (+2 ms)Writing region close event to WAL at 1676346834680 (+1 ms)Running coprocessor post-close hooks at 1676346834689 (+9 ms)Closed at 1676346834689 2023-02-14 03:53:54,690 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:54,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:54,691 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:54,694 DEBUG [M:0;jenkins-hbase9:41061] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2445e6e37b574aa087d5428803903794 as hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2445e6e37b574aa087d5428803903794 2023-02-14 03:53:54,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/.tmp/info/d2d886a8bd7e4d22bd29330c4a2bbca9 as hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info/d2d886a8bd7e4d22bd29330c4a2bbca9 2023-02-14 03:53:54,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info/d2d886a8bd7e4d22bd29330c4a2bbca9, entries=2, sequenceid=6, filesize=4.8 K 2023-02-14 03:53:54,703 INFO [M:0;jenkins-hbase9:41061] regionserver.HStore(1080): Added hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2445e6e37b574aa087d5428803903794, entries=17, sequenceid=141, filesize=8.5 K 2023-02-14 03:53:54,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for ee39c662a4152a296031f22713f8127e in 67ms, sequenceid=6, compaction requested=false 2023-02-14 03:53:54,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-14 03:53:54,704 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegion(2947): Finished flush of dataSize ~64.18 KB/65719, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 73ms, sequenceid=141, compaction requested=false 2023-02-14 03:53:54,707 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:53:54,707 DEBUG [M:0;jenkins-hbase9:41061] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676346834629Disabling compacts and flushes for region at 1676346834629Disabling writes for close at 1676346834631 (+2 ms)Obtaining lock to block concurrent updates at 1676346834632 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676346834632Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65719, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676346834632Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676346834635 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676346834639 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676346834652 (+13 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676346834652Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676346834695 (+43 ms)Finished flush of dataSize ~64.18 KB/65719, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 73ms, sequenceid=141, compaction requested=false at 1676346834704 (+9 ms)Writing region close event to WAL at 1676346834707 (+3 ms)Closed at 1676346834707 2023-02-14 03:53:54,708 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/.tmp/table/d8add518f26d4a79a3a699e468cad68f 2023-02-14 03:53:54,712 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 not finished, retry = 0 2023-02-14 03:53:54,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-14 03:53:54,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:54,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for ee39c662a4152a296031f22713f8127e: Waiting for close lock at 1676346834635Running coprocessor pre-close hooks at 1676346834635Disabling compacts and flushes for region at 1676346834635Disabling writes for close at 1676346834636 (+1 ms)Obtaining lock to block concurrent updates at 1676346834636Preparing flush snapshotting stores in ee39c662a4152a296031f22713f8127e at 1676346834636Finished memstore snapshotting hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676346834636Flushing stores of hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. at 1676346834640 (+4 ms)Flushing ee39c662a4152a296031f22713f8127e/info: creating writer at 1676346834640Flushing ee39c662a4152a296031f22713f8127e/info: appending metadata at 1676346834656 (+16 ms)Flushing ee39c662a4152a296031f22713f8127e/info: closing flushed file at 1676346834656Flushing ee39c662a4152a296031f22713f8127e/info: reopening flushed file at 1676346834695 (+39 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for ee39c662a4152a296031f22713f8127e in 67ms, sequenceid=6, compaction requested=false at 1676346834703 (+8 ms)Writing region close event to WAL at 1676346834707 (+4 ms)Running coprocessor post-close hooks at 1676346834717 (+10 ms)Closed at 1676346834717 2023-02-14 03:53:54,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:54,720 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/.tmp/info/0664f2c4f1e94c53aa17bec03a18be22 as hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info/0664f2c4f1e94c53aa17bec03a18be22 2023-02-14 03:53:54,726 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info/0664f2c4f1e94c53aa17bec03a18be22, entries=40, sequenceid=24, filesize=9.5 K 2023-02-14 03:53:54,729 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/.tmp/table/d8add518f26d4a79a3a699e468cad68f as hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table/d8add518f26d4a79a3a699e468cad68f 2023-02-14 03:53:54,735 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:54,735 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26b22500 rejected from java.util.concurrent.ThreadPoolExecutor@228da50e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:54,736 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:54,736 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:54,735 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table/d8add518f26d4a79a3a699e468cad68f, entries=8, sequenceid=24, filesize=5.0 K 2023-02-14 03:53:54,736 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:54,736 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:54,736 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@618f1b8a rejected from java.util.concurrent.ThreadPoolExecutor@228da50e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:54,736 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346830702 2023-02-14 03:53:54,737 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:54,738 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 98ms, sequenceid=24, compaction requested=false 2023-02-14 03:53:54,738 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-14 03:53:54,750 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-14 03:53:54,751 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-14 03:53:54,751 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 03:53:54,751 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676346834640Running coprocessor pre-close hooks at 1676346834640Disabling compacts and flushes for region at 1676346834640Disabling writes for close at 1676346834640Obtaining lock to block concurrent updates at 1676346834641 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676346834641Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676346834641Flushing stores of hbase:meta,,1.1588230740 at 1676346834641Flushing 1588230740/info: creating writer at 1676346834642 (+1 ms)Flushing 1588230740/info: appending metadata at 1676346834652 (+10 ms)Flushing 1588230740/info: closing flushed file at 1676346834652Flushing 1588230740/table: creating writer at 1676346834694 (+42 ms)Flushing 1588230740/table: appending metadata at 1676346834697 (+3 ms)Flushing 1588230740/table: closing flushed file at 1676346834697Flushing 1588230740/info: reopening flushed file at 1676346834721 (+24 ms)Flushing 1588230740/table: reopening flushed file at 1676346834730 (+9 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 98ms, sequenceid=24, compaction requested=false at 1676346834738 (+8 ms)Writing region close event to WAL at 1676346834744 (+6 ms)Running coprocessor post-close hooks at 1676346834751 (+7 ms)Closed at 1676346834751 2023-02-14 03:53:54,751 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-14 03:53:54,813 INFO [M:0;jenkins-hbase9:41061] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-14 03:53:54,813 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:54,814 INFO [M:0;jenkins-hbase9:41061] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:41061 2023-02-14 03:53:54,820 DEBUG [M:0;jenkins-hbase9:41061] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,41061,1676346830650 already deleted, retry=false 2023-02-14 03:53:54,837 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,33231,1676346830786; all regions closed. 2023-02-14 03:53:54,840 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,33009,1676346830739; all regions closed. 2023-02-14 03:53:54,844 DEBUG [RS:2;jenkins-hbase9:33231] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:53:54,844 INFO [RS:2;jenkins-hbase9:33231] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33231%2C1676346830786:(num 1676346831401) 2023-02-14 03:53:54,844 DEBUG [RS:2;jenkins-hbase9:33231] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,844 INFO [RS:2;jenkins-hbase9:33231] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:54,845 INFO [RS:2;jenkins-hbase9:33231] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:54,845 INFO [RS:2;jenkins-hbase9:33231] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:53:54,845 INFO [RS:2;jenkins-hbase9:33231] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:53:54,845 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:54,845 INFO [RS:2;jenkins-hbase9:33231] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:53:54,845 INFO [RS:2;jenkins-hbase9:33231] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33231 2023-02-14 03:53:54,849 DEBUG [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:53:54,849 INFO [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33009%2C1676346830739.meta:.meta(num 1676346832005) 2023-02-14 03:53:54,850 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:54,850 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:54,851 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@574974b7 rejected from java.util.concurrent.ThreadPoolExecutor@5376fa86[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:54,855 DEBUG [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:53:54,855 INFO [RS:1;jenkins-hbase9:33009] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33009%2C1676346830739:(num 1676346831422) 2023-02-14 03:53:54,855 DEBUG [RS:1;jenkins-hbase9:33009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:53:54,855 INFO [RS:1;jenkins-hbase9:33009] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:53:54,855 INFO [RS:1;jenkins-hbase9:33009] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:53:54,856 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:53:54,856 INFO [RS:1;jenkins-hbase9:33009] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33009 2023-02-14 03:53:54,858 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:54,858 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6b9a8aaa rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:54,930 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:54,930 INFO [M:0;jenkins-hbase9:41061] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,41061,1676346830650; zookeeper connection closed. 2023-02-14 03:53:54,930 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@22a16139 rejected from java.util.concurrent.ThreadPoolExecutor@269ee1b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:54,931 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:41061-0x10050f78f070000, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:54,931 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b34f516 rejected from java.util.concurrent.ThreadPoolExecutor@269ee1b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,030 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:55,030 INFO [RS:0;jenkins-hbase9:33791] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,33791,1676346830702; zookeeper connection closed. 2023-02-14 03:53:55,031 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1948c291 rejected from java.util.concurrent.ThreadPoolExecutor@228da50e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,031 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@17a5227c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@17a5227c 2023-02-14 03:53:55,031 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f070001, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:55,031 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1eead841 rejected from java.util.concurrent.ThreadPoolExecutor@228da50e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,331 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:55,331 INFO [RS:1;jenkins-hbase9:33009] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,33009,1676346830739; zookeeper connection closed. 2023-02-14 03:53:55,331 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24b7fb44 rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,332 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f070002, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:55,332 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5314ab9b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5314ab9b 2023-02-14 03:53:55,332 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5cbdcfd0 rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,431 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:55,431 INFO [RS:2;jenkins-hbase9:33231] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,33231,1676346830786; zookeeper connection closed. 2023-02-14 03:53:55,432 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3bd85c95 rejected from java.util.concurrent.ThreadPoolExecutor@5376fa86[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,432 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@59fb4ab8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@59fb4ab8 2023-02-14 03:53:55,432 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f070003, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:53:55,432 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@436e173c rejected from java.util.concurrent.ThreadPoolExecutor@5376fa86[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:53:55,432 INFO [Listener at localhost/46385] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-14 03:53:55,432 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(93): Sleeping a bit 2023-02-14 03:53:57,432 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(96): Starting cluster the second time with the same ports 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:57,445 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:57,446 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:38493 2023-02-14 03:53:57,447 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,448 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,450 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38493 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:57,453 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:384930x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:57,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38493-0x10050f78f070009 connected 2023-02-14 03:53:57,458 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:53:57,458 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-14 03:53:57,459 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:57,460 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38493 2023-02-14 03:53:57,460 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38493 2023-02-14 03:53:57,460 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38493 2023-02-14 03:53:57,460 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38493 2023-02-14 03:53:57,461 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38493 2023-02-14 03:53:57,461 INFO [Listener at localhost/46385] master.HMaster(439): hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99, hbase.cluster.distributed=false 2023-02-14 03:53:57,462 DEBUG [pool-503-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-14 03:53:57,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,470 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:57,470 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,471 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-14 03:53:57,471 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:53:57,472 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:57,472 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,38493,1676346837444 from backup master directory 2023-02-14 03:53:57,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:57,474 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,474 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:57,474 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:57,474 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,474 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-14 03:53:57,476 INFO [Listener at localhost/46385] client.ConnectionUtils(127): regionserver/jenkins-hbase9:33791 server-side Connection retries=45 2023-02-14 03:53:57,477 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,477 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,477 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:57,477 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,477 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:57,478 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:57,479 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33791 2023-02-14 03:53:57,479 INFO [Listener at localhost/46385] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:57,480 DEBUG [Listener at localhost/46385] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:57,481 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,482 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,483 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33791 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:57,489 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:57,490 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:57,491 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-14 03:53:57,492 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:337910x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:57,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33791-0x10050f78f07000a connected 2023-02-14 03:53:57,495 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33791 2023-02-14 03:53:57,496 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33791 2023-02-14 03:53:57,496 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33791 2023-02-14 03:53:57,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,500 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33791 2023-02-14 03:53:57,500 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33791 2023-02-14 03:53:57,502 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:57,502 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:57,516 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:57,516 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:57,526 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:57,528 DEBUG [RS:3;jenkins-hbase9:33791] zookeeper.ReadOnlyZKClient(139): Connect 0x7ec560c5 to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:57,536 DEBUG [RS:3;jenkins-hbase9:33791] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7002816d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:57,537 DEBUG [RS:3;jenkins-hbase9:33791] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@743d0bb5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:57,542 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2df98882 to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:57,548 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase9:33791 2023-02-14 03:53:57,549 INFO [RS:3;jenkins-hbase9:33791] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:57,549 INFO [RS:3;jenkins-hbase9:33791] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:57,549 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:57,549 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,38493,1676346837444 with isa=jenkins-hbase9.apache.org/172.31.2.10:33791, startcode=1676346837476 2023-02-14 03:53:57,549 DEBUG [RS:3;jenkins-hbase9:33791] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:57,552 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:39743, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.9 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:57,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:57,552 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@148f98d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:57,553 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-14 03:53:57,553 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 03:53:57,553 WARN [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-14 03:53:57,554 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-14 03:53:57,554 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:57,560 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650 to hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650-dead as it is dead 2023-02-14 03:53:57,560 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650-dead/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 2023-02-14 03:53:57,561 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650-dead/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 after 1ms 2023-02-14 03:53:57,561 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650-dead/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 to hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 2023-02-14 03:53:57,561 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,41061,1676346830650-dead 2023-02-14 03:53:57,562 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,564 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C38493%2C1676346837444, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,38493,1676346837444, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/oldWALs, maxLogs=10 2023-02-14 03:53:57,576 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:57,577 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:57,577 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:57,580 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/WALs/jenkins-hbase9.apache.org,38493,1676346837444/jenkins-hbase9.apache.org%2C38493%2C1676346837444.1676346837564 2023-02-14 03:53:57,580 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK], DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK]] 2023-02-14 03:53:57,580 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:57,580 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:57,580 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:57,580 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:57,582 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:57,583 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-14 03:53:57,584 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-14 03:53:57,590 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2445e6e37b574aa087d5428803903794 2023-02-14 03:53:57,591 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:57,591 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-14 03:53:57,591 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 2023-02-14 03:53:57,599 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 2023-02-14 03:53:57,600 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C41061%2C1676346830650.1676346830981 2023-02-14 03:53:57,605 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-14 03:53:57,612 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-14 03:53:57,613 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63353989, jitterRate=-0.05595199763774872}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:57,613 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676346837580Initializing all the Stores at 1676346837581 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346837581Cleaning up temporary data from old regions at 1676346837600 (+19 ms)Cleaning up detritus from prior splits at 1676346837602 (+2 ms)Region opened successfully at 1676346837613 (+11 ms) 2023-02-14 03:53:57,615 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-14 03:53:57,616 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-14 03:53:57,616 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-14 03:53:57,617 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-14 03:53:57,617 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-14 03:53:57,623 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-14 03:53:57,624 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-14 03:53:57,624 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-14 03:53:57,624 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-14 03:53:57,624 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-14 03:53:57,625 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-14 03:53:57,625 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-14 03:53:57,625 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-02-14 03:53:57,625 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-14 03:53:57,626 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-14 03:53:57,627 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33009,1676346830739, table=hbase:meta, region=1588230740 2023-02-14 03:53:57,629 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-14 03:53:57,630 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33231,1676346830786 already deleted, retry=false 2023-02-14 03:53:57,630 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,33231,1676346830786 on jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,632 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,33231,1676346830786, splitWal=true, meta=false 2023-02-14 03:53:57,632 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase9.apache.org,33231,1676346830786 (carryingMeta=false) jenkins-hbase9.apache.org,33231,1676346830786/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@3b92d9f0[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-14 03:53:57,634 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33009,1676346830739 already deleted, retry=false 2023-02-14 03:53:57,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,33009,1676346830739 on jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,636 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,33009,1676346830739, splitWal=true, meta=true 2023-02-14 03:53:57,636 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase9.apache.org,33009,1676346830739 (carryingMeta=true) jenkins-hbase9.apache.org,33009,1676346830739/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@644c6c42[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-14 03:53:57,637 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33791,1676346830702 already deleted, retry=false 2023-02-14 03:53:57,637 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,33791,1676346830702 on jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,639 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,33791,1676346830702, splitWal=true, meta=false 2023-02-14 03:53:57,640 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=20 for jenkins-hbase9.apache.org,33791,1676346830702 (carryingMeta=false) jenkins-hbase9.apache.org,33791,1676346830702/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@ebab801[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-14 03:53:57,651 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-14 03:53:57,652 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-14 03:53:57,652 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-14 03:53:57,653 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-14 03:53:57,653 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-14 03:53:57,654 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,38493,1676346837444 with isa=jenkins-hbase9.apache.org/172.31.2.10:33791, startcode=1676346837476 2023-02-14 03:53:57,654 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-14 03:53:57,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:57,654 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-14 03:53:57,655 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(2829): Master is not running yet 2023-02-14 03:53:57,655 WARN [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-14 03:53:57,655 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-14 03:53:57,655 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase9.apache.org,38493,1676346837444, sessionid=0x10050f78f070009, setting cluster-up flag (Was=true) 2023-02-14 03:53:57,658 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-14 03:53:57,659 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,661 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-14 03:53:57,662 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:53:57,663 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/.hbase-snapshot/.tmp 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:57,664 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,669 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676346867669 2023-02-14 03:53:57,669 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-14 03:53:57,669 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-14 03:53:57,670 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-14 03:53:57,670 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-14 03:53:57,670 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-14 03:53:57,670 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-14 03:53:57,671 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,671 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-14 03:53:57,672 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33009,1676346830739; numProcessing=1 2023-02-14 03:53:57,672 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-14 03:53:57,672 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33009,1676346830739, splitWal=true, meta=true 2023-02-14 03:53:57,672 DEBUG [PEWorker-3] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33791,1676346830702; numProcessing=2 2023-02-14 03:53:57,672 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-14 03:53:57,672 DEBUG [PEWorker-2] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33231,1676346830786; numProcessing=3 2023-02-14 03:53:57,672 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33231,1676346830786, splitWal=true, meta=false 2023-02-14 03:53:57,672 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=20, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33791,1676346830702, splitWal=true, meta=false 2023-02-14 03:53:57,673 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-14 03:53:57,673 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-14 03:53:57,673 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346837673,5,FailOnTimeoutGroup] 2023-02-14 03:53:57,674 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346837674,5,FailOnTimeoutGroup] 2023-02-14 03:53:57,674 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,674 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-14 03:53:57,674 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,674 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,674 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676346837470, completionTime=-1 2023-02-14 03:53:57,674 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=0; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-14 03:53:57,674 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33009,1676346830739, splitWal=true, meta=true, isMeta: true 2023-02-14 03:53:57,676 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739-splitting 2023-02-14 03:53:57,678 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739-splitting dir is empty, no logs to split. 2023-02-14 03:53:57,678 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase9.apache.org,33009,1676346830739 WAL count=0, meta=true 2023-02-14 03:53:57,680 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739-splitting dir is empty, no logs to split. 2023-02-14 03:53:57,680 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase9.apache.org,33009,1676346830739 WAL count=0, meta=true 2023-02-14 03:53:57,680 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33009,1676346830739 WAL splitting is done? wals=0, meta=true 2023-02-14 03:53:57,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-14 03:53:57,682 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-14 03:53:57,684 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=false 2023-02-14 03:53:57,834 WARN [jenkins-hbase9:38493] assignment.AssignmentManager(2157): No servers available; cannot place 1 unassigned regions. 2023-02-14 03:53:57,856 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,38493,1676346837444 with isa=jenkins-hbase9.apache.org/172.31.2.10:33791, startcode=1676346837476 2023-02-14 03:53:57,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,858 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:57,858 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42255 2023-02-14 03:53:57,858 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:57,859 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:57,860 DEBUG [RS:3;jenkins-hbase9:33791] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,860 WARN [RS:3;jenkins-hbase9:33791] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:57,860 INFO [RS:3;jenkins-hbase9:33791] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:57,861 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,861 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33791,1676346837476] 2023-02-14 03:53:57,865 DEBUG [RS:3;jenkins-hbase9:33791] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,866 DEBUG [RS:3;jenkins-hbase9:33791] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:57,866 INFO [RS:3;jenkins-hbase9:33791] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:57,868 INFO [RS:3;jenkins-hbase9:33791] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:57,868 INFO [RS:3;jenkins-hbase9:33791] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:57,868 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,868 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:57,870 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:33791, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,870 DEBUG [RS:3;jenkins-hbase9:33791] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:33791, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,872 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,872 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,872 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,874 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=1; waited=200ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-14 03:53:57,887 INFO [RS:3;jenkins-hbase9:33791] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:57,887 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33791,1676346837476-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,903 INFO [RS:3;jenkins-hbase9:33791] regionserver.Replication(203): jenkins-hbase9.apache.org,33791,1676346837476 started 2023-02-14 03:53:57,903 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,33791,1676346837476, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33791, sessionid=0x10050f78f07000a 2023-02-14 03:53:57,903 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:57,903 DEBUG [RS:3;jenkins-hbase9:33791] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,903 DEBUG [RS:3;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33791,1676346837476' 2023-02-14 03:53:57,904 DEBUG [RS:3;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:57,904 DEBUG [RS:3;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:57,905 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:57,905 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:57,905 DEBUG [RS:3;jenkins-hbase9:33791] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,905 DEBUG [RS:3;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33791,1676346837476' 2023-02-14 03:53:57,905 DEBUG [RS:3;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:57,905 DEBUG [RS:3;jenkins-hbase9:33791] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:57,906 DEBUG [RS:3;jenkins-hbase9:33791] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:57,906 INFO [RS:3;jenkins-hbase9:33791] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:57,906 INFO [RS:3;jenkins-hbase9:33791] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:57,917 INFO [Listener at localhost/46385] client.ConnectionUtils(127): regionserver/jenkins-hbase9:33009 server-side Connection retries=45 2023-02-14 03:53:57,917 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,917 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,917 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:57,918 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:57,918 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:57,918 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:57,919 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33009 2023-02-14 03:53:57,919 INFO [Listener at localhost/46385] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:57,921 DEBUG [Listener at localhost/46385] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:57,921 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,922 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:57,924 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33009 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:57,928 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:330090x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:57,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33009-0x10050f78f07000d connected 2023-02-14 03:53:57,930 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:57,931 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-14 03:53:57,932 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:57,935 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33009 2023-02-14 03:53:57,935 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33009 2023-02-14 03:53:57,936 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33009 2023-02-14 03:53:57,937 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33009 2023-02-14 03:53:57,937 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33009 2023-02-14 03:53:57,939 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:57,939 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:57,941 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:57,941 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:57,943 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:57,944 DEBUG [RS:4;jenkins-hbase9:33009] zookeeper.ReadOnlyZKClient(139): Connect 0x2686135f to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:57,950 DEBUG [RS:4;jenkins-hbase9:33009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2841e9ff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:57,951 DEBUG [RS:4;jenkins-hbase9:33009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c7e782f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:57,959 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:4;jenkins-hbase9:33009 2023-02-14 03:53:57,959 INFO [RS:4;jenkins-hbase9:33009] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:57,959 INFO [RS:4;jenkins-hbase9:33009] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:57,959 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:57,960 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,38493,1676346837444 with isa=jenkins-hbase9.apache.org/172.31.2.10:33009, startcode=1676346837916 2023-02-14 03:53:57,960 DEBUG [RS:4;jenkins-hbase9:33009] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:57,962 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54825, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.10 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:57,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:57,963 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:57,963 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42255 2023-02-14 03:53:57,963 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:57,965 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:57,965 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:57,965 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33009,1676346837916] 2023-02-14 03:53:57,965 DEBUG [RS:4;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:57,965 WARN [RS:4;jenkins-hbase9:33009] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:57,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:57,965 INFO [RS:4;jenkins-hbase9:33009] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:57,966 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:57,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,970 DEBUG [RS:4;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:57,970 DEBUG [RS:4;jenkins-hbase9:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:57,971 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:57,971 INFO [RS:4;jenkins-hbase9:33009] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:57,973 INFO [RS:4;jenkins-hbase9:33009] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:57,975 INFO [RS:4;jenkins-hbase9:33009] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:57,975 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,975 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=301ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-14 03:53:57,976 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:57,977 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,977 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:33009, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,978 DEBUG [RS:4;jenkins-hbase9:33009] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:33009, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:57,983 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,983 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,983 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:57,994 INFO [RS:4;jenkins-hbase9:33009] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:57,994 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33009,1676346837916-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,005 INFO [RS:4;jenkins-hbase9:33009] regionserver.Replication(203): jenkins-hbase9.apache.org,33009,1676346837916 started 2023-02-14 03:53:58,005 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,33009,1676346837916, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33009, sessionid=0x10050f78f07000d 2023-02-14 03:53:58,005 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:58,005 DEBUG [RS:4;jenkins-hbase9:33009] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,005 DEBUG [RS:4;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33009,1676346837916' 2023-02-14 03:53:58,005 DEBUG [RS:4;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:58,005 DEBUG [RS:4;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:58,006 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:58,006 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:58,006 DEBUG [RS:4;jenkins-hbase9:33009] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,006 DEBUG [RS:4;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33009,1676346837916' 2023-02-14 03:53:58,006 DEBUG [RS:4;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:58,006 DEBUG [RS:4;jenkins-hbase9:33009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:58,007 DEBUG [RS:4;jenkins-hbase9:33009] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:58,007 INFO [RS:4;jenkins-hbase9:33009] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:58,007 INFO [RS:4;jenkins-hbase9:33009] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:58,008 INFO [RS:3;jenkins-hbase9:33791] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33791%2C1676346837476, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346837476, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:58,016 INFO [Listener at localhost/46385] client.ConnectionUtils(127): regionserver/jenkins-hbase9:33231 server-side Connection retries=45 2023-02-14 03:53:58,016 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:58,016 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:58,016 INFO [Listener at localhost/46385] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-14 03:53:58,016 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-14 03:53:58,017 INFO [Listener at localhost/46385] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-14 03:53:58,017 INFO [Listener at localhost/46385] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-14 03:53:58,017 INFO [Listener at localhost/46385] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33231 2023-02-14 03:53:58,018 INFO [Listener at localhost/46385] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-14 03:53:58,019 DEBUG [Listener at localhost/46385] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-14 03:53:58,021 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:58,022 INFO [Listener at localhost/46385] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-14 03:53:58,024 INFO [Listener at localhost/46385] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33231 connecting to ZooKeeper ensemble=127.0.0.1:59369 2023-02-14 03:53:58,024 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:58,024 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:58,026 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:58,029 INFO [RS:3;jenkins-hbase9:33791] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346837476/jenkins-hbase9.apache.org%2C33791%2C1676346837476.1676346838008 2023-02-14 03:53:58,029 DEBUG [RS:3;jenkins-hbase9:33791] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK]] 2023-02-14 03:53:58,030 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:332310x0, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-14 03:53:58,031 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): regionserver:332310x0, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-14 03:53:58,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33231-0x10050f78f07000f connected 2023-02-14 03:53:58,032 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-14 03:53:58,033 DEBUG [Listener at localhost/46385] zookeeper.ZKUtil(164): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-14 03:53:58,036 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33231 2023-02-14 03:53:58,037 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33231 2023-02-14 03:53:58,039 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33231 2023-02-14 03:53:58,040 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33231 2023-02-14 03:53:58,041 DEBUG [Listener at localhost/46385] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33231 2023-02-14 03:53:58,043 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(952): ClusterId : ed080027-b8d6-4c2b-bffa-437c7d74b026 2023-02-14 03:53:58,043 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-14 03:53:58,045 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-14 03:53:58,046 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-14 03:53:58,047 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-14 03:53:58,048 DEBUG [RS:5;jenkins-hbase9:33231] zookeeper.ReadOnlyZKClient(139): Connect 0x75304a94 to 127.0.0.1:59369 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-14 03:53:58,053 DEBUG [RS:5;jenkins-hbase9:33231] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a59db3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-14 03:53:58,053 DEBUG [RS:5;jenkins-hbase9:33231] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@448ff8e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:53:58,062 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:5;jenkins-hbase9:33231 2023-02-14 03:53:58,062 INFO [RS:5;jenkins-hbase9:33231] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-14 03:53:58,062 INFO [RS:5;jenkins-hbase9:33231] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-14 03:53:58,062 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1023): About to register with Master. 2023-02-14 03:53:58,062 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,38493,1676346837444 with isa=jenkins-hbase9.apache.org/172.31.2.10:33231, startcode=1676346838016 2023-02-14 03:53:58,062 DEBUG [RS:5;jenkins-hbase9:33231] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-14 03:53:58,067 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:57135, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.11 (auth:SIMPLE), service=RegionServerStatusService 2023-02-14 03:53:58,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,068 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99 2023-02-14 03:53:58,068 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42255 2023-02-14 03:53:58,068 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-14 03:53:58,070 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:58,070 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:58,070 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:53:58,070 DEBUG [RS:5;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,070 WARN [RS:5;jenkins-hbase9:33231] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-14 03:53:58,070 INFO [RS:5;jenkins-hbase9:33231] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:58,070 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33231,1676346838016] 2023-02-14 03:53:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,071 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:58,076 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=401ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-14 03:53:58,076 WARN [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1676346837684, server=jenkins-hbase9.apache.org,33009,1676346830739}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-14 03:53:58,078 DEBUG [RS:5;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,078 DEBUG [RS:5;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,078 DEBUG [RS:5;jenkins-hbase9:33231] zookeeper.ZKUtil(162): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:53:58,079 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-14 03:53:58,080 INFO [RS:5;jenkins-hbase9:33231] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-14 03:53:58,084 INFO [RS:5;jenkins-hbase9:33231] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-14 03:53:58,084 INFO [jenkins-hbase9:38493] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-14 03:53:58,085 INFO [RS:5;jenkins-hbase9:33231] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-14 03:53:58,085 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33009,1676346837916, state=OPENING 2023-02-14 03:53:58,085 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,085 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-14 03:53:58,087 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,088 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-14 03:53:58,088 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 03:53:58,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33009,1676346837916}] 2023-02-14 03:53:58,088 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,088 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,088 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,089 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,095 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,095 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:33231, corePoolSize=2, maxPoolSize=2 2023-02-14 03:53:58,096 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,097 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,097 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,097 DEBUG [RS:5;jenkins-hbase9:33231] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:33231, corePoolSize=1, maxPoolSize=1 2023-02-14 03:53:58,099 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,099 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,100 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,109 INFO [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33009%2C1676346837916, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346837916, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:58,113 INFO [RS:5;jenkins-hbase9:33231] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-14 03:53:58,113 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33231,1676346838016-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:58,124 DEBUG [RS-EventLoopGroup-15-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:58,124 DEBUG [RS-EventLoopGroup-15-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:58,124 DEBUG [RS-EventLoopGroup-15-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:58,129 INFO [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346837916/jenkins-hbase9.apache.org%2C33009%2C1676346837916.1676346838109 2023-02-14 03:53:58,131 DEBUG [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK], DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK]] 2023-02-14 03:53:58,132 INFO [RS:5;jenkins-hbase9:33231] regionserver.Replication(203): jenkins-hbase9.apache.org,33231,1676346838016 started 2023-02-14 03:53:58,132 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,33231,1676346838016, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33231, sessionid=0x10050f78f07000f 2023-02-14 03:53:58,132 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-14 03:53:58,132 DEBUG [RS:5;jenkins-hbase9:33231] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,132 DEBUG [RS:5;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33231,1676346838016' 2023-02-14 03:53:58,132 DEBUG [RS:5;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-14 03:53:58,132 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:53:58,133 DEBUG [RS:5;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-14 03:53:58,133 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-14 03:53:58,133 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-14 03:53:58,133 DEBUG [RS:5;jenkins-hbase9:33231] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:58,133 DEBUG [RS:5;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33231,1676346838016' 2023-02-14 03:53:58,133 DEBUG [RS:5;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-14 03:53:58,134 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53024, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-14 03:53:58,134 DEBUG [RS:5;jenkins-hbase9:33231] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-14 03:53:58,134 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33009] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.2.10:53024 deadline: 1676346898134, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,134 DEBUG [RS:5;jenkins-hbase9:33231] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-14 03:53:58,134 INFO [RS:5;jenkins-hbase9:33231] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-14 03:53:58,134 INFO [RS:5;jenkins-hbase9:33231] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-14 03:53:58,236 WARN [ReadOnlyZKClient-127.0.0.1:59369@0x4b95821d] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-02-14 03:53:58,236 INFO [RS:5;jenkins-hbase9:33231] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33231%2C1676346838016, suffix=, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346838016, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:58,237 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33009] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Get size: 90 connection: 172.31.2.10:53024 deadline: 1676346898236, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,261 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:58,261 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:58,261 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:58,262 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:58,262 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:58,263 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53026, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-14 03:53:58,267 INFO [RS:5;jenkins-hbase9:33231] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346838016/jenkins-hbase9.apache.org%2C33231%2C1676346838016.1676346838237 2023-02-14 03:53:58,269 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-14 03:53:58,270 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-14 03:53:58,270 DEBUG [RS:5;jenkins-hbase9:33231] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK]] 2023-02-14 03:53:58,272 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33009%2C1676346837916.meta, suffix=.meta, logDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346837916, archiveDir=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs, maxLogs=32 2023-02-14 03:53:58,288 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK] 2023-02-14 03:53:58,288 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK] 2023-02-14 03:53:58,289 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK] 2023-02-14 03:53:58,291 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346837916/jenkins-hbase9.apache.org%2C33009%2C1676346837916.meta.1676346838272.meta 2023-02-14 03:53:58,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37243,DS-8ed84e9a-5657-4856-9cb1-c51063fb72df,DISK], DatanodeInfoWithStorage[127.0.0.1:43537,DS-3b80a63f-92bc-4bb5-8ed8-c3a1f95c106f,DISK], DatanodeInfoWithStorage[127.0.0.1:35271,DS-a4e18589-5411-42b9-becb-3384a440deb7,DISK]] 2023-02-14 03:53:58,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:58,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-14 03:53:58,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-14 03:53:58,292 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-14 03:53:58,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-14 03:53:58,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:58,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-14 03:53:58,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-14 03:53:58,294 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-14 03:53:58,296 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info 2023-02-14 03:53:58,296 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info 2023-02-14 03:53:58,296 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-14 03:53:58,304 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info/0664f2c4f1e94c53aa17bec03a18be22 2023-02-14 03:53:58,304 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:58,305 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-14 03:53:58,306 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:58,306 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/rep_barrier 2023-02-14 03:53:58,306 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-14 03:53:58,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:58,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-14 03:53:58,308 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table 2023-02-14 03:53:58,308 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table 2023-02-14 03:53:58,309 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-14 03:53:58,320 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/table/d8add518f26d4a79a3a699e468cad68f 2023-02-14 03:53:58,320 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:58,323 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740 2023-02-14 03:53:58,324 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740 2023-02-14 03:53:58,327 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-14 03:53:58,329 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-14 03:53:58,330 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59943907, jitterRate=-0.10676617920398712}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-14 03:53:58,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676346838292Writing region info on filesystem at 1676346838292Initializing all the Stores at 1676346838293 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346838293Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346838294 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346838294Cleaning up temporary data from old regions at 1676346838325 (+31 ms)Cleaning up detritus from prior splits at 1676346838326 (+1 ms)Running coprocessor post-open hooks at 1676346838330 (+4 ms)Region opened successfully at 1676346838330 2023-02-14 03:53:58,334 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=22, masterSystemTime=1676346838262 2023-02-14 03:53:58,339 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-14 03:53:58,339 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-14 03:53:58,340 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33009,1676346837916, state=OPEN 2023-02-14 03:53:58,342 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-14 03:53:58,343 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-14 03:53:58,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-14 03:53:58,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33009,1676346837916 in 255 msec 2023-02-14 03:53:58,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=19 2023-02-14 03:53:58,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 665 msec 2023-02-14 03:53:58,454 WARN [RPCClient-NioEventLoopGroup-6-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:41061 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:41061 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:58,455 WARN [Listener at localhost/46385] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase9.apache.org/172.31.2.10:41061 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:41061 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:41061 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-14 03:53:58,455 DEBUG [RPCClient-NioEventLoopGroup-6-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:41061 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:41061 2023-02-14 03:53:58,456 DEBUG [Listener at localhost/46385] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:41061 this server is in the failed servers list 2023-02-14 03:53:58,456 WARN [Listener at localhost/46385] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase9.apache.org/172.31.2.10:41061 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase9.apache.org/172.31.2.10:41061 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnClusterRestart(TestRetainAssignmentOnRestart.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase9.apache.org/172.31.2.10:41061 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-14 03:53:58,457 DEBUG [Listener at localhost/46385] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-14 03:53:58,459 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34434, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-14 03:53:58,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.CallRunner(144): callId: 50 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.2.10:34434 deadline: 1676346898459, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-14 03:53:58,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:58,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.CallRunner(144): callId: 52 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.2.10:34434 deadline: 1676346898561, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-14 03:53:58,630 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-14 03:53:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.CallRunner(144): callId: 54 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.2.10:34434 deadline: 1676346898764, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-14 03:53:58,821 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-14 03:53:58,821 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-14 03:53:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.CallRunner(144): callId: 56 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.2.10:34434 deadline: 1676346899068, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-14 03:53:59,077 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-14 03:53:59,082 DEBUG [hconnection-0x12a114ae-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:53:59,084 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53038, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-14 03:53:59,087 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=ee39c662a4152a296031f22713f8127e, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33231,1676346830786, regionLocation=jenkins-hbase9.apache.org,33231,1676346830786, openSeqNum=2 2023-02-14 03:53:59,088 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=a93dd5cbcf0c0282a4de644346a8b9ec, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33009,1676346830739, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739, openSeqNum=2 2023-02-14 03:53:59,088 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=bd6d9dbbd3807b958cb89c661120405d, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33009,1676346830739, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739, openSeqNum=2 2023-02-14 03:53:59,088 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=1da7aa25683c42faa92bcbdf37a68280, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33009,1676346830739, regionLocation=jenkins-hbase9.apache.org,33009,1676346830739, openSeqNum=2 2023-02-14 03:53:59,089 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-14 03:53:59,089 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676346899089 2023-02-14 03:53:59,089 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676346959089 2023-02-14 03:53:59,089 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 12 msec 2023-02-14 03:53:59,112 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,33009,1676346830739 had 4 regions 2023-02-14 03:53:59,112 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,33231,1676346830786 had 1 regions 2023-02-14 03:53:59,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38493,1676346837444-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:59,112 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,33791,1676346830702 had 0 regions 2023-02-14 03:53:59,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38493,1676346837444-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:59,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38493,1676346837444-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:59,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:38493, period=300000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:59,112 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-14 03:53:59,113 WARN [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1230): hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. is NOT online; state={ee39c662a4152a296031f22713f8127e state=OPEN, ts=1676346839087, server=jenkins-hbase9.apache.org,33231,1676346830786}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-14 03:53:59,113 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33231,1676346830786, splitWal=true, meta=false, isMeta: false 2023-02-14 03:53:59,113 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33791,1676346830702, splitWal=true, meta=false, isMeta: false 2023-02-14 03:53:59,114 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33009,1676346830739, splitWal=true, meta=true, isMeta: false 2023-02-14 03:53:59,118 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786-splitting 2023-02-14 03:53:59,118 WARN [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase9.apache.org,33231,1676346830786/hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e., unknown_server=jenkins-hbase9.apache.org,33009,1676346830739/restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec., unknown_server=jenkins-hbase9.apache.org,33009,1676346830739/restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d., unknown_server=jenkins-hbase9.apache.org,33009,1676346830739/restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:59,119 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786-splitting dir is empty, no logs to split. 2023-02-14 03:53:59,119 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase9.apache.org,33231,1676346830786 WAL count=0, meta=false 2023-02-14 03:53:59,120 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702-splitting 2023-02-14 03:53:59,121 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702-splitting dir is empty, no logs to split. 2023-02-14 03:53:59,122 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase9.apache.org,33791,1676346830702 WAL count=0, meta=false 2023-02-14 03:53:59,123 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739-splitting dir is empty, no logs to split. 2023-02-14 03:53:59,123 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase9.apache.org,33009,1676346830739 WAL count=0, meta=false 2023-02-14 03:53:59,125 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786-splitting dir is empty, no logs to split. 2023-02-14 03:53:59,125 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase9.apache.org,33231,1676346830786 WAL count=0, meta=false 2023-02-14 03:53:59,125 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33231,1676346830786 WAL splitting is done? wals=0, meta=false 2023-02-14 03:53:59,127 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702-splitting dir is empty, no logs to split. 2023-02-14 03:53:59,127 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase9.apache.org,33791,1676346830702 WAL count=0, meta=false 2023-02-14 03:53:59,127 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33791,1676346830702 WAL splitting is done? wals=0, meta=false 2023-02-14 03:53:59,127 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33009,1676346830739-splitting dir is empty, no logs to split. 2023-02-14 03:53:59,127 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase9.apache.org,33009,1676346830739 WAL count=0, meta=false 2023-02-14 03:53:59,127 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33009,1676346830739 WAL splitting is done? wals=0, meta=false 2023-02-14 03:53:59,128 INFO [PEWorker-3] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase9.apache.org,33231,1676346830786 failed, ignore...File hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33231,1676346830786-splitting does not exist. 2023-02-14 03:53:59,129 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase9.apache.org,33791,1676346830702 failed, ignore...File hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/WALs/jenkins-hbase9.apache.org,33791,1676346830702-splitting does not exist. 2023-02-14 03:53:59,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN}] 2023-02-14 03:53:59,130 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN}, {pid=25, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN}, {pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN}] 2023-02-14 03:53:59,131 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN 2023-02-14 03:53:59,131 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,33791,1676346830702 after splitting done 2023-02-14 03:53:59,131 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase9.apache.org,33791,1676346830702 from processing; numProcessing=2 2023-02-14 03:53:59,131 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN 2023-02-14 03:53:59,132 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN 2023-02-14 03:53:59,132 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN 2023-02-14 03:53:59,133 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,33231,1676346830786 2023-02-14 03:53:59,133 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,33231,1676346830786; forceNewPlan=false, retain=true 2023-02-14 03:53:59,134 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:59,134 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:59,134 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=true 2023-02-14 03:53:59,134 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=true 2023-02-14 03:53:59,134 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,33009,1676346830739 2023-02-14 03:53:59,134 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN; state=OPEN, location=jenkins-hbase9.apache.org,33009,1676346830739; forceNewPlan=false, retain=true 2023-02-14 03:53:59,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,33791,1676346830702, splitWal=true, meta=false in 1.4940 sec 2023-02-14 03:53:59,284 INFO [jenkins-hbase9:38493] balancer.BaseLoadBalancer(1545): Reassigned 4 regions. 4 retained the pre-restart assignment. 2023-02-14 03:53:59,287 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=a93dd5cbcf0c0282a4de644346a8b9ec, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:59,287 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=1da7aa25683c42faa92bcbdf37a68280, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:59,287 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346839287"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346839287"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346839287"}]},"ts":"1676346839287"} 2023-02-14 03:53:59,287 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=bd6d9dbbd3807b958cb89c661120405d, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:59,287 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346839287"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346839287"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346839287"}]},"ts":"1676346839287"} 2023-02-14 03:53:59,287 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346839287"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346839287"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346839287"}]},"ts":"1676346839287"} 2023-02-14 03:53:59,287 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=ee39c662a4152a296031f22713f8127e, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:59,287 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346839287"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676346839287"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676346839287"}]},"ts":"1676346839287"} 2023-02-14 03:53:59,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure a93dd5cbcf0c0282a4de644346a8b9ec, server=jenkins-hbase9.apache.org,33009,1676346837916}] 2023-02-14 03:53:59,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=25, state=RUNNABLE; OpenRegionProcedure 1da7aa25683c42faa92bcbdf37a68280, server=jenkins-hbase9.apache.org,33009,1676346837916}] 2023-02-14 03:53:59,292 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=26, state=RUNNABLE; OpenRegionProcedure bd6d9dbbd3807b958cb89c661120405d, server=jenkins-hbase9.apache.org,33009,1676346837916}] 2023-02-14 03:53:59,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE; OpenRegionProcedure ee39c662a4152a296031f22713f8127e, server=jenkins-hbase9.apache.org,33231,1676346838016}] 2023-02-14 03:53:59,447 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:59,447 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-14 03:53:59,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:59,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1da7aa25683c42faa92bcbdf37a68280, NAME => 'restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:59,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:59,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7890): checking encryption for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7893): checking classloading for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,449 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54512, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-14 03:53:59,450 INFO [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,451 DEBUG [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/family 2023-02-14 03:53:59,451 DEBUG [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/family 2023-02-14 03:53:59,452 INFO [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1da7aa25683c42faa92bcbdf37a68280 columnFamilyName family 2023-02-14 03:53:59,453 INFO [StoreOpener-1da7aa25683c42faa92bcbdf37a68280-1] regionserver.HStore(310): Store=1da7aa25683c42faa92bcbdf37a68280/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:59,453 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:59,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee39c662a4152a296031f22713f8127e, NAME => 'hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:59,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:59,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(7890): checking encryption for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(7893): checking classloading for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,455 INFO [StoreOpener-ee39c662a4152a296031f22713f8127e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,456 DEBUG [StoreOpener-ee39c662a4152a296031f22713f8127e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info 2023-02-14 03:53:59,456 DEBUG [StoreOpener-ee39c662a4152a296031f22713f8127e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info 2023-02-14 03:53:59,456 INFO [StoreOpener-ee39c662a4152a296031f22713f8127e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee39c662a4152a296031f22713f8127e columnFamilyName info 2023-02-14 03:53:59,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1054): writing seq id for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:53:59,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1071): Opened 1da7aa25683c42faa92bcbdf37a68280; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65466626, jitterRate=-0.024471253156661987}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:59,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(964): Region open journal for 1da7aa25683c42faa92bcbdf37a68280: Running coprocessor pre-open hook at 1676346839449Writing region info on filesystem at 1676346839449Initializing all the Stores at 1676346839450 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346839450Cleaning up temporary data from old regions at 1676346839455 (+5 ms)Cleaning up detritus from prior splits at 1676346839456 (+1 ms)Running coprocessor post-open hooks at 1676346839459 (+3 ms)Region opened successfully at 1676346839459 2023-02-14 03:53:59,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280., pid=28, masterSystemTime=1676346839443 2023-02-14 03:53:59,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:59,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:53:59,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:59,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd6d9dbbd3807b958cb89c661120405d, NAME => 'restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:59,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,464 DEBUG [StoreOpener-ee39c662a4152a296031f22713f8127e-1] regionserver.HStore(539): loaded hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/info/d2d886a8bd7e4d22bd29330c4a2bbca9 2023-02-14 03:53:59,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:59,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7890): checking encryption for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,464 INFO [StoreOpener-ee39c662a4152a296031f22713f8127e-1] regionserver.HStore(310): Store=ee39c662a4152a296031f22713f8127e/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:59,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7893): checking classloading for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,464 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=1da7aa25683c42faa92bcbdf37a68280, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:59,465 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346839464"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346839464"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346839464"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346839464"}]},"ts":"1676346839464"} 2023-02-14 03:53:59,465 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,465 INFO [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,466 DEBUG [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/family 2023-02-14 03:53:59,466 DEBUG [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/family 2023-02-14 03:53:59,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,467 INFO [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd6d9dbbd3807b958cb89c661120405d columnFamilyName family 2023-02-14 03:53:59,468 INFO [StoreOpener-bd6d9dbbd3807b958cb89c661120405d-1] regionserver.HStore(310): Store=bd6d9dbbd3807b958cb89c661120405d/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:59,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=25 2023-02-14 03:53:59,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=25, state=SUCCESS; OpenRegionProcedure 1da7aa25683c42faa92bcbdf37a68280, server=jenkins-hbase9.apache.org,33009,1676346837916 in 175 msec 2023-02-14 03:53:59,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,470 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1054): writing seq id for ee39c662a4152a296031f22713f8127e 2023-02-14 03:53:59,471 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=1da7aa25683c42faa92bcbdf37a68280, ASSIGN in 339 msec 2023-02-14 03:53:59,472 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1071): Opened ee39c662a4152a296031f22713f8127e; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58924275, jitterRate=-0.1219598799943924}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:59,472 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(964): Region open journal for ee39c662a4152a296031f22713f8127e: Running coprocessor pre-open hook at 1676346839453Writing region info on filesystem at 1676346839453Initializing all the Stores at 1676346839454 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676346839455 (+1 ms)Cleaning up temporary data from old regions at 1676346839467 (+12 ms)Cleaning up detritus from prior splits at 1676346839468 (+1 ms)Running coprocessor post-open hooks at 1676346839472 (+4 ms)Region opened successfully at 1676346839472 2023-02-14 03:53:59,472 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e., pid=30, masterSystemTime=1676346839447 2023-02-14 03:53:59,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1054): writing seq id for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:53:59,475 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:59,476 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:33231-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:53:59,476 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=ee39c662a4152a296031f22713f8127e, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:53:59,476 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676346839476"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346839476"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346839476"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346839476"}]},"ts":"1676346839476"} 2023-02-14 03:53:59,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1071): Opened bd6d9dbbd3807b958cb89c661120405d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68072939, jitterRate=0.01436583697795868}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:59,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(964): Region open journal for bd6d9dbbd3807b958cb89c661120405d: Running coprocessor pre-open hook at 1676346839464Writing region info on filesystem at 1676346839464Initializing all the Stores at 1676346839465 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346839465Cleaning up temporary data from old regions at 1676346839471 (+6 ms)Cleaning up detritus from prior splits at 1676346839471Running coprocessor post-open hooks at 1676346839477 (+6 ms)Region opened successfully at 1676346839477 2023-02-14 03:53:59,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d., pid=29, masterSystemTime=1676346839443 2023-02-14 03:53:59,479 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:59,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:53:59,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:59,479 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7850): Opening region: {ENCODED => a93dd5cbcf0c0282a4de644346a8b9ec, NAME => 'restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.', STARTKEY => '', ENDKEY => ''} 2023-02-14 03:53:59,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-14 03:53:59,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7890): checking encryption for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,480 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=bd6d9dbbd3807b958cb89c661120405d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:59,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(7893): checking classloading for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,480 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676346839480"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346839480"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346839480"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346839480"}]},"ts":"1676346839480"} 2023-02-14 03:53:59,481 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-02-14 03:53:59,481 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; OpenRegionProcedure ee39c662a4152a296031f22713f8127e, server=jenkins-hbase9.apache.org,33231,1676346838016 in 185 msec 2023-02-14 03:53:59,482 INFO [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,484 DEBUG [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/family 2023-02-14 03:53:59,484 DEBUG [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/family 2023-02-14 03:53:59,484 INFO [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a93dd5cbcf0c0282a4de644346a8b9ec columnFamilyName family 2023-02-14 03:53:59,486 INFO [StoreOpener-a93dd5cbcf0c0282a4de644346a8b9ec-1] regionserver.HStore(310): Store=a93dd5cbcf0c0282a4de644346a8b9ec/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-14 03:53:59,486 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=18 2023-02-14 03:53:59,487 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,33231,1676346830786 after splitting done 2023-02-14 03:53:59,487 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase9.apache.org,33231,1676346830786 from processing; numProcessing=1 2023-02-14 03:53:59,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ee39c662a4152a296031f22713f8127e, ASSIGN in 352 msec 2023-02-14 03:53:59,487 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=26 2023-02-14 03:53:59,487 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=26, state=SUCCESS; OpenRegionProcedure bd6d9dbbd3807b958cb89c661120405d, server=jenkins-hbase9.apache.org,33009,1676346837916 in 190 msec 2023-02-14 03:53:59,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=bd6d9dbbd3807b958cb89c661120405d, ASSIGN in 357 msec 2023-02-14 03:53:59,489 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,33231,1676346830786, splitWal=true, meta=false in 1.8560 sec 2023-02-14 03:53:59,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1054): writing seq id for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:53:59,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1071): Opened a93dd5cbcf0c0282a4de644346a8b9ec; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68768106, jitterRate=0.024724632501602173}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-14 03:53:59,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(964): Region open journal for a93dd5cbcf0c0282a4de644346a8b9ec: Running coprocessor pre-open hook at 1676346839480Writing region info on filesystem at 1676346839480Initializing all the Stores at 1676346839482 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676346839482Cleaning up temporary data from old regions at 1676346839490 (+8 ms)Cleaning up detritus from prior splits at 1676346839490Running coprocessor post-open hooks at 1676346839495 (+5 ms)Region opened successfully at 1676346839495 2023-02-14 03:53:59,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec., pid=27, masterSystemTime=1676346839443 2023-02-14 03:53:59,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:59,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:33009-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:53:59,503 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=a93dd5cbcf0c0282a4de644346a8b9ec, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:53:59,503 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676346839503"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676346839503"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676346839503"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676346839503"}]},"ts":"1676346839503"} 2023-02-14 03:53:59,513 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-02-14 03:53:59,513 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure a93dd5cbcf0c0282a4de644346a8b9ec, server=jenkins-hbase9.apache.org,33009,1676346837916 in 217 msec 2023-02-14 03:53:59,516 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=19 2023-02-14 03:53:59,516 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,33009,1676346830739 after splitting done 2023-02-14 03:53:59,516 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=a93dd5cbcf0c0282a4de644346a8b9ec, ASSIGN in 383 msec 2023-02-14 03:53:59,516 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase9.apache.org,33009,1676346830739 from processing; numProcessing=0 2023-02-14 03:53:59,519 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,33009,1676346830739, splitWal=true, meta=true in 1.8830 sec 2023-02-14 03:53:59,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-14 03:53:59,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38493] ipc.CallRunner(144): callId: 58 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.2.10:34434 deadline: 1676346899570, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-14 03:54:00,113 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-14 03:54:00,119 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-14 03:54:00,120 INFO [RS-EventLoopGroup-16-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54520, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-14 03:54:00,134 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-14 03:54:00,138 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-14 03:54:00,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.664sec 2023-02-14 03:54:00,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-14 03:54:00,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-14 03:54:00,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-14 03:54:00,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38493,1676346837444-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-14 03:54:00,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38493,1676346837444-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-14 03:54:00,142 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-14 03:54:00,586 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:54:00,603 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:54:00,610 INFO [Listener at localhost/46385] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-14 03:54:00,611 INFO [Listener at localhost/46385] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-14 03:54:00,621 INFO [Listener at localhost/46385] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-14 03:54:00,621 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => a93dd5cbcf0c0282a4de644346a8b9ec, NAME => 'restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,33009,1676346830739, currentServer=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:54:00,621 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 1da7aa25683c42faa92bcbdf37a68280, NAME => 'restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,33009,1676346830739, currentServer=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:54:00,621 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => bd6d9dbbd3807b958cb89c661120405d, NAME => 'restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,33009,1676346830739, currentServer=jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:54:00,622 INFO [Listener at localhost/46385] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => ee39c662a4152a296031f22713f8127e, NAME => 'hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase9.apache.org,33231,1676346830786, currentServer=jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:54:00,622 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-14 03:54:00,622 INFO [Listener at localhost/46385] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-14 03:54:00,623 DEBUG [Listener at localhost/46385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b95821d to 127.0.0.1:59369 2023-02-14 03:54:00,624 DEBUG [Listener at localhost/46385] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,624 DEBUG [Listener at localhost/46385] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-14 03:54:00,624 DEBUG [Listener at localhost/46385] util.JVMClusterUtil(257): Found active master hash=965614952, stopped=false 2023-02-14 03:54:00,625 INFO [Listener at localhost/46385] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:54:00,626 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:54:00,626 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:54:00,626 INFO [Listener at localhost/46385] procedure2.ProcedureExecutor(629): Stopping 2023-02-14 03:54:00,626 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:54:00,626 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-14 03:54:00,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:54:00,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:54:00,627 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:54:00,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:54:00,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-14 03:54:00,630 DEBUG [Listener at localhost/46385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2df98882 to 127.0.0.1:59369 2023-02-14 03:54:00,630 DEBUG [Listener at localhost/46385] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,631 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,33791,1676346837476' ***** 2023-02-14 03:54:00,631 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:54:00,631 INFO [RS:3;jenkins-hbase9:33791] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:54:00,631 INFO [RS:3;jenkins-hbase9:33791] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:54:00,631 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:54:00,631 INFO [RS:3;jenkins-hbase9:33791] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:54:00,632 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:54:00,632 DEBUG [RS:3;jenkins-hbase9:33791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ec560c5 to 127.0.0.1:59369 2023-02-14 03:54:00,632 DEBUG [RS:3;jenkins-hbase9:33791] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,632 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,33791,1676346837476; all regions closed. 2023-02-14 03:54:00,632 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,33009,1676346837916' ***** 2023-02-14 03:54:00,632 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:54:00,632 INFO [Listener at localhost/46385] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,33231,1676346838016' ***** 2023-02-14 03:54:00,632 INFO [Listener at localhost/46385] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-14 03:54:00,639 INFO [RS:5;jenkins-hbase9:33231] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:54:00,640 INFO [RS:4;jenkins-hbase9:33009] regionserver.HeapMemoryManager(220): Stopping 2023-02-14 03:54:00,640 INFO [RS:5;jenkins-hbase9:33231] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:54:00,640 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:54:00,640 INFO [RS:5;jenkins-hbase9:33231] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:54:00,640 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-14 03:54:00,642 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(3304): Received CLOSE for ee39c662a4152a296031f22713f8127e 2023-02-14 03:54:00,640 INFO [RS:4;jenkins-hbase9:33009] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-14 03:54:00,643 INFO [RS:4;jenkins-hbase9:33009] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-14 03:54:00,643 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for a93dd5cbcf0c0282a4de644346a8b9ec 2023-02-14 03:54:00,651 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:54:00,651 DEBUG [RS:5;jenkins-hbase9:33231] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75304a94 to 127.0.0.1:59369 2023-02-14 03:54:00,651 DEBUG [RS:5;jenkins-hbase9:33231] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,651 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-14 03:54:00,651 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1479): Online Regions={ee39c662a4152a296031f22713f8127e=hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e.} 2023-02-14 03:54:00,651 DEBUG [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1505): Waiting on ee39c662a4152a296031f22713f8127e 2023-02-14 03:54:00,655 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:54:00,661 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for 1da7aa25683c42faa92bcbdf37a68280 2023-02-14 03:54:00,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1603): Closing a93dd5cbcf0c0282a4de644346a8b9ec, disabling compactions & flushes 2023-02-14 03:54:00,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:54:00,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:54:00,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. after waiting 0 ms 2023-02-14 03:54:00,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:54:00,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1603): Closing ee39c662a4152a296031f22713f8127e, disabling compactions & flushes 2023-02-14 03:54:00,661 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:54:00,662 DEBUG [RS:4;jenkins-hbase9:33009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2686135f to 127.0.0.1:59369 2023-02-14 03:54:00,662 DEBUG [RS:4;jenkins-hbase9:33009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,662 INFO [RS:4;jenkins-hbase9:33009] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:54:00,662 INFO [RS:4;jenkins-hbase9:33009] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:54:00,662 INFO [RS:4;jenkins-hbase9:33009] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:54:00,662 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-14 03:54:00,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:54:00,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:54:00,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. after waiting 0 ms 2023-02-14 03:54:00,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:54:00,667 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1475): Waiting on 4 regions to close 2023-02-14 03:54:00,667 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1479): Online Regions={a93dd5cbcf0c0282a4de644346a8b9ec=restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec., 1588230740=hbase:meta,,1.1588230740, bd6d9dbbd3807b958cb89c661120405d=restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d., 1da7aa25683c42faa92bcbdf37a68280=restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280.} 2023-02-14 03:54:00,667 DEBUG [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1505): Waiting on 1588230740, 1da7aa25683c42faa92bcbdf37a68280, a93dd5cbcf0c0282a4de644346a8b9ec, bd6d9dbbd3807b958cb89c661120405d 2023-02-14 03:54:00,669 DEBUG [RS:3;jenkins-hbase9:33791] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:54:00,669 INFO [RS:3;jenkins-hbase9:33791] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33791%2C1676346837476:(num 1676346838008) 2023-02-14 03:54:00,669 DEBUG [RS:3;jenkins-hbase9:33791] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,669 INFO [RS:3;jenkins-hbase9:33791] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:54:00,674 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-14 03:54:00,674 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-14 03:54:00,674 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-14 03:54:00,674 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-14 03:54:00,674 INFO [RS:3;jenkins-hbase9:33791] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:33791 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:54:00,674 INFO [RS:3;jenkins-hbase9:33791] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:54:00,674 INFO [RS:3;jenkins-hbase9:33791] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:54:00,674 INFO [RS:3;jenkins-hbase9:33791] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:54:00,675 INFO [RS:3;jenkins-hbase9:33791] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33791 2023-02-14 03:54:00,674 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-14 03:54:00,676 INFO [regionserver/jenkins-hbase9:33791.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:54:00,674 INFO [regionserver/jenkins-hbase9:33791.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:54:00,678 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=3.81 KB heapSize=6.91 KB 2023-02-14 03:54:00,686 INFO [regionserver/jenkins-hbase9:33009.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:54:00,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableOne/a93dd5cbcf0c0282a4de644346a8b9ec/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-14 03:54:00,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1837): Closed restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:54:00,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1557): Region close journal for a93dd5cbcf0c0282a4de644346a8b9ec: Waiting for close lock at 1676346840658Running coprocessor pre-close hooks at 1676346840658Disabling compacts and flushes for region at 1676346840658Disabling writes for close at 1676346840661 (+3 ms)Writing region close event to WAL at 1676346840674 (+13 ms)Running coprocessor post-close hooks at 1676346840692 (+18 ms)Closed at 1676346840692 2023-02-14 03:54:00,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676346832762.a93dd5cbcf0c0282a4de644346a8b9ec. 2023-02-14 03:54:00,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1603): Closing bd6d9dbbd3807b958cb89c661120405d, disabling compactions & flushes 2023-02-14 03:54:00,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:54:00,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:54:00,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. after waiting 0 ms 2023-02-14 03:54:00,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:54:00,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/namespace/ee39c662a4152a296031f22713f8127e/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-02-14 03:54:00,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:54:00,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] regionserver.HRegion(1557): Region close journal for ee39c662a4152a296031f22713f8127e: Waiting for close lock at 1676346840655Running coprocessor pre-close hooks at 1676346840655Disabling compacts and flushes for region at 1676346840655Disabling writes for close at 1676346840663 (+8 ms)Writing region close event to WAL at 1676346840680 (+17 ms)Running coprocessor post-close hooks at 1676346840697 (+17 ms)Closed at 1676346840697 2023-02-14 03:54:00,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33231-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676346832115.ee39c662a4152a296031f22713f8127e. 2023-02-14 03:54:00,704 INFO [regionserver/jenkins-hbase9:33231.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:54:00,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableThree/bd6d9dbbd3807b958cb89c661120405d/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-14 03:54:00,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1837): Closed restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:54:00,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1557): Region close journal for bd6d9dbbd3807b958cb89c661120405d: Waiting for close lock at 1676346840692Running coprocessor pre-close hooks at 1676346840692Disabling compacts and flushes for region at 1676346840692Disabling writes for close at 1676346840693 (+1 ms)Writing region close event to WAL at 1676346840697 (+4 ms)Running coprocessor post-close hooks at 1676346840715 (+18 ms)Closed at 1676346840715 2023-02-14 03:54:00,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676346833995.bd6d9dbbd3807b958cb89c661120405d. 2023-02-14 03:54:00,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1603): Closing 1da7aa25683c42faa92bcbdf37a68280, disabling compactions & flushes 2023-02-14 03:54:00,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:54:00,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:54:00,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. after waiting 0 ms 2023-02-14 03:54:00,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:54:00,728 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.81 KB at sequenceid=38 (bloomFilter=false), to=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/.tmp/info/1ca0d63255b44efe85597ecc33f065ac 2023-02-14 03:54:00,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/default/restartTableTwo/1da7aa25683c42faa92bcbdf37a68280/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-14 03:54:00,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:54:00,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1557): Region close journal for 1da7aa25683c42faa92bcbdf37a68280: Waiting for close lock at 1676346840716Running coprocessor pre-close hooks at 1676346840716Disabling compacts and flushes for region at 1676346840716Disabling writes for close at 1676346840718 (+2 ms)Writing region close event to WAL at 1676346840723 (+5 ms)Running coprocessor post-close hooks at 1676346840730 (+7 ms)Closed at 1676346840730 2023-02-14 03:54:00,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:33009-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676346833380.1da7aa25683c42faa92bcbdf37a68280. 2023-02-14 03:54:00,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/.tmp/info/1ca0d63255b44efe85597ecc33f065ac as hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info/1ca0d63255b44efe85597ecc33f065ac 2023-02-14 03:54:00,744 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HStore(1080): Added hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/info/1ca0d63255b44efe85597ecc33f065ac, entries=32, sequenceid=38, filesize=8.5 K 2023-02-14 03:54:00,745 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(2947): Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 68ms, sequenceid=38, compaction requested=false 2023-02-14 03:54:00,755 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/data/hbase/meta/1588230740/recovered.edits/41.seqid, newMaxSeqId=41, maxSeqId=27 2023-02-14 03:54:00,756 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-14 03:54:00,757 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-14 03:54:00,757 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676346840667Running coprocessor pre-close hooks at 1676346840667Disabling compacts and flushes for region at 1676346840667Disabling writes for close at 1676346840674 (+7 ms)Obtaining lock to block concurrent updates at 1676346840678 (+4 ms)Preparing flush snapshotting stores in 1588230740 at 1676346840678Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=3900, getHeapSize=7032, getOffHeapSize=0, getCellsCount=32 at 1676346840679 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676346840680 (+1 ms)Flushing 1588230740/info: creating writer at 1676346840681 (+1 ms)Flushing 1588230740/info: appending metadata at 1676346840690 (+9 ms)Flushing 1588230740/info: closing flushed file at 1676346840690Flushing 1588230740/info: reopening flushed file at 1676346840738 (+48 ms)Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 68ms, sequenceid=38, compaction requested=false at 1676346840745 (+7 ms)Writing region close event to WAL at 1676346840751 (+6 ms)Running coprocessor post-close hooks at 1676346840756 (+5 ms)Closed at 1676346840757 (+1 ms) 2023-02-14 03:54:00,757 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:33009-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-14 03:54:00,766 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:54:00,767 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@775f3422 rejected from java.util.concurrent.ThreadPoolExecutor@7360be11[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:00,767 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:54:00,767 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@96ed669 rejected from java.util.concurrent.ThreadPoolExecutor@7360be11[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:00,767 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:54:00,767 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33791,1676346837476 2023-02-14 03:54:00,767 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:54:00,767 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:54:00,767 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:54:00,770 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33791,1676346837476] 2023-02-14 03:54:00,770 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33791,1676346837476; numProcessing=1 2023-02-14 03:54:00,772 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33791,1676346837476 already deleted, retry=false 2023-02-14 03:54:00,772 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,33791,1676346837476 expired; onlineServers=2 2023-02-14 03:54:00,851 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,33231,1676346838016; all regions closed. 2023-02-14 03:54:00,858 DEBUG [RS:5;jenkins-hbase9:33231] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:54:00,858 INFO [RS:5;jenkins-hbase9:33231] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33231%2C1676346838016:(num 1676346838237) 2023-02-14 03:54:00,858 DEBUG [RS:5;jenkins-hbase9:33231] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,858 INFO [RS:5;jenkins-hbase9:33231] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:54:00,858 INFO [RS:5;jenkins-hbase9:33231] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:33231 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:54:00,858 INFO [RS:5;jenkins-hbase9:33231] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-14 03:54:00,858 INFO [regionserver/jenkins-hbase9:33231.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:54:00,858 INFO [RS:5;jenkins-hbase9:33231] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-14 03:54:00,858 INFO [RS:5;jenkins-hbase9:33231] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-14 03:54:00,859 INFO [RS:5;jenkins-hbase9:33231] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33231 2023-02-14 03:54:00,862 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:54:00,862 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:54:00,862 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33231,1676346838016 2023-02-14 03:54:00,862 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33231,1676346838016] 2023-02-14 03:54:00,863 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33231,1676346838016; numProcessing=2 2023-02-14 03:54:00,864 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33231,1676346838016 already deleted, retry=false 2023-02-14 03:54:00,864 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,33231,1676346838016 expired; onlineServers=1 2023-02-14 03:54:00,867 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,33009,1676346837916; all regions closed. 2023-02-14 03:54:00,873 DEBUG [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:54:00,873 INFO [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33009%2C1676346837916.meta:.meta(num 1676346838272) 2023-02-14 03:54:00,879 DEBUG [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/oldWALs 2023-02-14 03:54:00,879 INFO [RS:4;jenkins-hbase9:33009] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33009%2C1676346837916:(num 1676346838109) 2023-02-14 03:54:00,880 DEBUG [RS:4;jenkins-hbase9:33009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,880 INFO [RS:4;jenkins-hbase9:33009] regionserver.LeaseManager(133): Closed leases 2023-02-14 03:54:00,880 INFO [RS:4;jenkins-hbase9:33009] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:33009 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-14 03:54:00,880 INFO [regionserver/jenkins-hbase9:33009.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:54:00,881 INFO [RS:4;jenkins-hbase9:33009] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33009 2023-02-14 03:54:00,883 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33009,1676346837916 2023-02-14 03:54:00,883 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-14 03:54:00,883 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@338e1579 rejected from java.util.concurrent.ThreadPoolExecutor@37b187d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:00,884 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33009,1676346837916] 2023-02-14 03:54:00,884 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33009,1676346837916; numProcessing=3 2023-02-14 03:54:00,885 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33009,1676346837916 already deleted, retry=false 2023-02-14 03:54:00,885 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,33009,1676346837916 expired; onlineServers=0 2023-02-14 03:54:00,885 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,38493,1676346837444' ***** 2023-02-14 03:54:00,885 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-14 03:54:00,886 DEBUG [M:1;jenkins-hbase9:38493] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3db19e81, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-14 03:54:00,886 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,38493,1676346837444 2023-02-14 03:54:00,886 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,38493,1676346837444; all regions closed. 2023-02-14 03:54:00,886 DEBUG [M:1;jenkins-hbase9:38493] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-14 03:54:00,887 DEBUG [M:1;jenkins-hbase9:38493] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-14 03:54:00,887 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-14 03:54:00,887 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346837674] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1676346837674,5,FailOnTimeoutGroup] 2023-02-14 03:54:00,887 DEBUG [M:1;jenkins-hbase9:38493] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-14 03:54:00,887 INFO [M:1;jenkins-hbase9:38493] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-14 03:54:00,887 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346837673] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1676346837673,5,FailOnTimeoutGroup] 2023-02-14 03:54:00,888 INFO [M:1;jenkins-hbase9:38493] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-14 03:54:00,888 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-14 03:54:00,888 INFO [M:1;jenkins-hbase9:38493] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-02-14 03:54:00,888 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-14 03:54:00,889 DEBUG [M:1;jenkins-hbase9:38493] master.HMaster(1502): Stopping service threads 2023-02-14 03:54:00,889 INFO [M:1;jenkins-hbase9:38493] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-14 03:54:00,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-14 03:54:00,889 ERROR [M:1;jenkins-hbase9:38493] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-14 03:54:00,889 INFO [M:1;jenkins-hbase9:38493] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-14 03:54:00,889 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-14 03:54:00,890 DEBUG [M:1;jenkins-hbase9:38493] zookeeper.ZKUtil(398): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-14 03:54:00,890 WARN [M:1;jenkins-hbase9:38493] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-14 03:54:00,890 INFO [M:1;jenkins-hbase9:38493] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-14 03:54:00,890 INFO [M:1;jenkins-hbase9:38493] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-14 03:54:00,891 DEBUG [M:1;jenkins-hbase9:38493] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-14 03:54:00,891 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:54:00,891 DEBUG [M:1;jenkins-hbase9:38493] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:54:00,891 DEBUG [M:1;jenkins-hbase9:38493] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-14 03:54:00,891 DEBUG [M:1;jenkins-hbase9:38493] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:54:00,891 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.74 KB heapSize=47.95 KB 2023-02-14 03:54:00,904 INFO [M:1;jenkins-hbase9:38493] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.74 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0e04f21e1d9b40318143676868e3f701 2023-02-14 03:54:00,909 DEBUG [M:1;jenkins-hbase9:38493] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0e04f21e1d9b40318143676868e3f701 as hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0e04f21e1d9b40318143676868e3f701 2023-02-14 03:54:00,914 INFO [M:1;jenkins-hbase9:38493] regionserver.HStore(1080): Added hdfs://localhost:42255/user/jenkins/test-data/d5dfa684-1cd1-7ad1-2b40-6c515d67cc99/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0e04f21e1d9b40318143676868e3f701, entries=13, sequenceid=246, filesize=6.2 K 2023-02-14 03:54:00,915 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegion(2947): Finished flush of dataSize ~38.74 KB/39667, heapSize ~47.93 KB/49080, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 24ms, sequenceid=246, compaction requested=false 2023-02-14 03:54:00,917 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-14 03:54:00,917 DEBUG [M:1;jenkins-hbase9:38493] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676346840891Disabling compacts and flushes for region at 1676346840891Disabling writes for close at 1676346840891Obtaining lock to block concurrent updates at 1676346840891Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676346840891Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=39667, getHeapSize=49080, getOffHeapSize=0, getCellsCount=121 at 1676346840891Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676346840892 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676346840893 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676346840896 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676346840896Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676346840910 (+14 ms)Finished flush of dataSize ~38.74 KB/39667, heapSize ~47.93 KB/49080, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 24ms, sequenceid=246, compaction requested=false at 1676346840915 (+5 ms)Writing region close event to WAL at 1676346840917 (+2 ms)Closed at 1676346840917 2023-02-14 03:54:00,920 INFO [M:1;jenkins-hbase9:38493] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-14 03:54:00,920 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-14 03:54:00,921 INFO [M:1;jenkins-hbase9:38493] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:38493 2023-02-14 03:54:00,924 DEBUG [M:1;jenkins-hbase9:38493] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,38493,1676346837444 already deleted, retry=false 2023-02-14 03:54:01,126 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,126 INFO [M:1;jenkins-hbase9:38493] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,38493,1676346837444; zookeeper connection closed. 2023-02-14 03:54:01,127 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@59f1418e rejected from java.util.concurrent.ThreadPoolExecutor@5dcece68[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,128 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): master:38493-0x10050f78f070009, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,128 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ab033d1 rejected from java.util.concurrent.ThreadPoolExecutor@5dcece68[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,227 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,227 INFO [RS:4;jenkins-hbase9:33009] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,33009,1676346837916; zookeeper connection closed. 2023-02-14 03:54:01,227 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@edc158 rejected from java.util.concurrent.ThreadPoolExecutor@37b187d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,227 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@8f396b4] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@8f396b4 2023-02-14 03:54:01,227 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10050f78f07000d, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,227 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78bfa957 rejected from java.util.concurrent.ThreadPoolExecutor@37b187d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,327 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,327 INFO [RS:5;jenkins-hbase9:33231] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,33231,1676346838016; zookeeper connection closed. 2023-02-14 03:54:01,327 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2de8492 rejected from java.util.concurrent.ThreadPoolExecutor@6e10011[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,327 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3239d2fb] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3239d2fb 2023-02-14 03:54:01,327 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33231-0x10050f78f07000f, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,327 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4e6a7d7 rejected from java.util.concurrent.ThreadPoolExecutor@6e10011[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,427 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,427 INFO [RS:3;jenkins-hbase9:33791] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,33791,1676346837476; zookeeper connection closed. 2023-02-14 03:54:01,427 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@71f8bb86 rejected from java.util.concurrent.ThreadPoolExecutor@7360be11[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,427 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@737c555a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@737c555a 2023-02-14 03:54:01,427 DEBUG [Listener at localhost/46385-EventThread] zookeeper.ZKWatcher(600): regionserver:33791-0x10050f78f07000a, quorum=127.0.0.1:59369, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-14 03:54:01,427 ERROR [Listener at localhost/46385-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@29ef9868 rejected from java.util.concurrent.ThreadPoolExecutor@7360be11[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-14 03:54:01,428 INFO [Listener at localhost/46385] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 6 regionserver(s) complete 2023-02-14 03:54:01,430 WARN [Listener at localhost/46385] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 03:54:01,433 INFO [Listener at localhost/46385] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:54:01,537 WARN [BP-1299339611-172.31.2.10-1676346829676 heartbeating to localhost/127.0.0.1:42255] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 03:54:01,538 WARN [BP-1299339611-172.31.2.10-1676346829676 heartbeating to localhost/127.0.0.1:42255] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1299339611-172.31.2.10-1676346829676 (Datanode Uuid a43aa0f6-e723-4569-86af-29691e5aa956) service to localhost/127.0.0.1:42255 2023-02-14 03:54:01,539 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/dfs/data/data5/current/BP-1299339611-172.31.2.10-1676346829676] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:54:01,539 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/dfs/data/data6/current/BP-1299339611-172.31.2.10-1676346829676] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:54:01,541 WARN [Listener at localhost/46385] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 03:54:01,549 INFO [Listener at localhost/46385] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:54:01,655 WARN [BP-1299339611-172.31.2.10-1676346829676 heartbeating to localhost/127.0.0.1:42255] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 03:54:01,656 WARN [BP-1299339611-172.31.2.10-1676346829676 heartbeating to localhost/127.0.0.1:42255] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1299339611-172.31.2.10-1676346829676 (Datanode Uuid 1e5929df-1979-4aff-b66f-e0e3c6547a97) service to localhost/127.0.0.1:42255 2023-02-14 03:54:01,657 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/dfs/data/data3/current/BP-1299339611-172.31.2.10-1676346829676] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:54:01,657 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/dfs/data/data4/current/BP-1299339611-172.31.2.10-1676346829676] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:54:01,658 WARN [Listener at localhost/46385] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-14 03:54:01,664 INFO [Listener at localhost/46385] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:54:01,767 WARN [BP-1299339611-172.31.2.10-1676346829676 heartbeating to localhost/127.0.0.1:42255] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-14 03:54:01,767 WARN [BP-1299339611-172.31.2.10-1676346829676 heartbeating to localhost/127.0.0.1:42255] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1299339611-172.31.2.10-1676346829676 (Datanode Uuid 5d365fbe-0c52-4882-bf12-29e6aef44dde) service to localhost/127.0.0.1:42255 2023-02-14 03:54:01,862 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/dfs/data/data1/current/BP-1299339611-172.31.2.10-1676346829676] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:54:01,863 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f09f477-98c6-90f7-707c-d98acb77a4ed/cluster_7ab75752-4c05-eb36-25b1-6542bc1482bd/dfs/data/data2/current/BP-1299339611-172.31.2.10-1676346829676] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-14 03:54:01,878 INFO [Listener at localhost/46385] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-14 03:54:01,900 INFO [Listener at localhost/46385] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-14 03:54:01,945 INFO [Listener at localhost/46385] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-14 03:54:01,969 INFO [Listener at localhost/46385] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=154 (was 101) Potentially hanging thread: RS-EventLoopGroup-13-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/46385 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:33791.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1701509641) connection to localhost/127.0.0.1:42255 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:33231.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:33009.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost:42255 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=603 (was 523) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 347), ProcessCount=171 (was 171), AvailableMemoryMB=2533 (was 3337)