2023-02-13 04:53:33,673 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart timeout: 13 mins 2023-02-13 04:53:33,675 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestartSplitWithoutZk timeout: 13 mins 2023-02-13 04:53:33,713 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=7, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=289, ProcessCount=168, AvailableMemoryMB=5365 2023-02-13 04:53:33,965 DEBUG [Time-limited test] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396 2023-02-13 04:53:33,966 INFO [Time-limited test] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 04:53:33,970 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 04:53:33,970 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43, deleteOnExit=true 2023-02-13 04:53:33,971 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 04:53:33,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/test.cache.data in system properties and HBase conf 2023-02-13 04:53:33,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 04:53:33,972 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/hadoop.log.dir in system properties and HBase conf 2023-02-13 04:53:33,972 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 04:53:33,973 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 04:53:33,973 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 04:53:34,079 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 04:53:34,480 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 04:53:34,485 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 04:53:34,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 04:53:34,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 04:53:34,487 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 04:53:34,487 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 04:53:34,487 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 04:53:34,488 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 04:53:34,488 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 04:53:34,488 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 04:53:34,489 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/nfs.dump.dir in system properties and HBase conf 2023-02-13 04:53:34,489 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/java.io.tmpdir in system properties and HBase conf 2023-02-13 04:53:34,490 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 04:53:34,490 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 04:53:34,491 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 04:53:35,069 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 04:53:35,074 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 04:53:35,364 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 04:53:35,541 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 04:53:35,556 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:35,596 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:35,629 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/java.io.tmpdir/Jetty_localhost_35807_hdfs____.vr6kl5/webapp 2023-02-13 04:53:35,801 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35807 2023-02-13 04:53:35,813 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 04:53:35,813 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 04:53:36,308 WARN [Listener at localhost/37257] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:36,433 WARN [Listener at localhost/37257] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 04:53:36,455 WARN [Listener at localhost/37257] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:36,461 INFO [Listener at localhost/37257] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:36,467 INFO [Listener at localhost/37257] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/java.io.tmpdir/Jetty_localhost_40027_datanode____xhcqu8/webapp 2023-02-13 04:53:36,584 INFO [Listener at localhost/37257] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40027 2023-02-13 04:53:36,997 WARN [Listener at localhost/33939] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:37,011 WARN [Listener at localhost/33939] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 04:53:37,016 WARN [Listener at localhost/33939] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:37,018 INFO [Listener at localhost/33939] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:37,030 INFO [Listener at localhost/33939] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/java.io.tmpdir/Jetty_localhost_44495_datanode____.keyyix/webapp 2023-02-13 04:53:37,165 INFO [Listener at localhost/33939] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44495 2023-02-13 04:53:37,183 WARN [Listener at localhost/34721] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:37,217 WARN [Listener at localhost/34721] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 04:53:37,222 WARN [Listener at localhost/34721] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:37,226 INFO [Listener at localhost/34721] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:37,242 INFO [Listener at localhost/34721] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/java.io.tmpdir/Jetty_localhost_45875_datanode____gk6hjw/webapp 2023-02-13 04:53:37,395 INFO [Listener at localhost/34721] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45875 2023-02-13 04:53:37,412 WARN [Listener at localhost/41311] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:37,635 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x181840c9e8d43eae: Processing first storage report for DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c from datanode 0c1e1f28-bae9-4953-b26e-014656334ec1 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x181840c9e8d43eae: from storage DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c node DatanodeRegistration(127.0.0.1:35517, datanodeUuid=0c1e1f28-bae9-4953-b26e-014656334ec1, infoPort=41993, infoSecurePort=0, ipcPort=33939, storageInfo=lv=-57;cid=testClusterID;nsid=381692484;c=1676264015148), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb98f003083486e69: Processing first storage report for DS-06b0245c-348a-46fe-a73a-e9588f517ee5 from datanode 4625348a-b30e-4cb4-ba12-9854c58ab386 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb98f003083486e69: from storage DS-06b0245c-348a-46fe-a73a-e9588f517ee5 node DatanodeRegistration(127.0.0.1:36071, datanodeUuid=4625348a-b30e-4cb4-ba12-9854c58ab386, infoPort=44229, infoSecurePort=0, ipcPort=41311, storageInfo=lv=-57;cid=testClusterID;nsid=381692484;c=1676264015148), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x79531f8c98a263a5: Processing first storage report for DS-40d161fd-6e75-44c9-bccd-3a1cbf486688 from datanode 6ed8807f-8601-499f-9051-a604c2a88c5d 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x79531f8c98a263a5: from storage DS-40d161fd-6e75-44c9-bccd-3a1cbf486688 node DatanodeRegistration(127.0.0.1:35683, datanodeUuid=6ed8807f-8601-499f-9051-a604c2a88c5d, infoPort=46435, infoSecurePort=0, ipcPort=34721, storageInfo=lv=-57;cid=testClusterID;nsid=381692484;c=1676264015148), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x181840c9e8d43eae: Processing first storage report for DS-7bf677e8-b298-491a-aa17-f4b7b9ed8013 from datanode 0c1e1f28-bae9-4953-b26e-014656334ec1 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x181840c9e8d43eae: from storage DS-7bf677e8-b298-491a-aa17-f4b7b9ed8013 node DatanodeRegistration(127.0.0.1:35517, datanodeUuid=0c1e1f28-bae9-4953-b26e-014656334ec1, infoPort=41993, infoSecurePort=0, ipcPort=33939, storageInfo=lv=-57;cid=testClusterID;nsid=381692484;c=1676264015148), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb98f003083486e69: Processing first storage report for DS-62ae7fa0-6ebe-469f-b1c2-baccc2ccbff8 from datanode 4625348a-b30e-4cb4-ba12-9854c58ab386 2023-02-13 04:53:37,637 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb98f003083486e69: from storage DS-62ae7fa0-6ebe-469f-b1c2-baccc2ccbff8 node DatanodeRegistration(127.0.0.1:36071, datanodeUuid=4625348a-b30e-4cb4-ba12-9854c58ab386, infoPort=44229, infoSecurePort=0, ipcPort=41311, storageInfo=lv=-57;cid=testClusterID;nsid=381692484;c=1676264015148), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:37,638 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x79531f8c98a263a5: Processing first storage report for DS-e8e740e6-9698-49f5-a811-56083f0dd4c2 from datanode 6ed8807f-8601-499f-9051-a604c2a88c5d 2023-02-13 04:53:37,638 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x79531f8c98a263a5: from storage DS-e8e740e6-9698-49f5-a811-56083f0dd4c2 node DatanodeRegistration(127.0.0.1:35683, datanodeUuid=6ed8807f-8601-499f-9051-a604c2a88c5d, infoPort=46435, infoSecurePort=0, ipcPort=34721, storageInfo=lv=-57;cid=testClusterID;nsid=381692484;c=1676264015148), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:37,879 DEBUG [Listener at localhost/41311] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396 2023-02-13 04:53:37,941 INFO [Listener at localhost/41311] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/zookeeper_0, clientPort=52493, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 04:53:37,953 INFO [Listener at localhost/41311] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52493 2023-02-13 04:53:37,961 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:37,962 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:38,612 INFO [Listener at localhost/41311] util.FSUtils(479): Created version file at hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c with version=8 2023-02-13 04:53:38,613 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/hbase-staging 2023-02-13 04:53:38,907 INFO [Listener at localhost/41311] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 04:53:39,386 INFO [Listener at localhost/41311] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:39,418 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:39,418 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:39,419 INFO [Listener at localhost/41311] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:39,419 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:39,419 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:39,565 INFO [Listener at localhost/41311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:39,646 DEBUG [Listener at localhost/41311] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 04:53:39,751 INFO [Listener at localhost/41311] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42441 2023-02-13 04:53:39,762 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:39,763 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:39,785 INFO [Listener at localhost/41311] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42441 connecting to ZooKeeper ensemble=127.0.0.1:52493 2023-02-13 04:53:39,826 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:424410x0, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:39,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:42441-0x1004c06f9cb0000 connected 2023-02-13 04:53:39,857 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:39,858 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:39,862 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:39,870 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42441 2023-02-13 04:53:39,870 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42441 2023-02-13 04:53:39,871 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42441 2023-02-13 04:53:39,871 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42441 2023-02-13 04:53:39,872 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42441 2023-02-13 04:53:39,878 INFO [Listener at localhost/41311] master.HMaster(439): hbase.rootdir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c, hbase.cluster.distributed=false 2023-02-13 04:53:39,949 INFO [Listener at localhost/41311] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:39,949 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:39,950 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:39,950 INFO [Listener at localhost/41311] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:39,950 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:39,950 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:39,955 INFO [Listener at localhost/41311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:39,958 INFO [Listener at localhost/41311] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33925 2023-02-13 04:53:39,960 INFO [Listener at localhost/41311] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:39,968 DEBUG [Listener at localhost/41311] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:39,970 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:39,972 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:39,975 INFO [Listener at localhost/41311] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33925 connecting to ZooKeeper ensemble=127.0.0.1:52493 2023-02-13 04:53:39,979 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:339250x0, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:39,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33925-0x1004c06f9cb0001 connected 2023-02-13 04:53:39,980 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:39,981 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:39,982 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:39,983 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33925 2023-02-13 04:53:39,983 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33925 2023-02-13 04:53:39,984 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33925 2023-02-13 04:53:39,986 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33925 2023-02-13 04:53:39,986 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33925 2023-02-13 04:53:40,003 INFO [Listener at localhost/41311] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:40,003 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:40,003 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:40,004 INFO [Listener at localhost/41311] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:40,004 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:40,004 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:40,004 INFO [Listener at localhost/41311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:40,006 INFO [Listener at localhost/41311] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36873 2023-02-13 04:53:40,006 INFO [Listener at localhost/41311] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:40,009 DEBUG [Listener at localhost/41311] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:40,009 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:40,012 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:40,014 INFO [Listener at localhost/41311] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36873 connecting to ZooKeeper ensemble=127.0.0.1:52493 2023-02-13 04:53:40,020 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:368730x0, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:40,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36873-0x1004c06f9cb0002 connected 2023-02-13 04:53:40,021 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:40,022 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:40,023 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:40,024 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36873 2023-02-13 04:53:40,024 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36873 2023-02-13 04:53:40,024 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36873 2023-02-13 04:53:40,025 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36873 2023-02-13 04:53:40,025 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36873 2023-02-13 04:53:40,037 INFO [Listener at localhost/41311] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:40,038 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:40,038 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:40,038 INFO [Listener at localhost/41311] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:40,038 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:40,038 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:40,038 INFO [Listener at localhost/41311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:40,040 INFO [Listener at localhost/41311] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37801 2023-02-13 04:53:40,040 INFO [Listener at localhost/41311] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:40,041 DEBUG [Listener at localhost/41311] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:40,042 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:40,043 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:40,045 INFO [Listener at localhost/41311] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37801 connecting to ZooKeeper ensemble=127.0.0.1:52493 2023-02-13 04:53:40,048 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:378010x0, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:40,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37801-0x1004c06f9cb0003 connected 2023-02-13 04:53:40,049 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:40,050 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:40,050 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:40,052 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37801 2023-02-13 04:53:40,052 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37801 2023-02-13 04:53:40,053 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37801 2023-02-13 04:53:40,054 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37801 2023-02-13 04:53:40,054 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37801 2023-02-13 04:53:40,057 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,066 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:40,068 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,086 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:40,087 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:40,087 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:40,086 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:40,086 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:40,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:40,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42441,1676264018749 from backup master directory 2023-02-13 04:53:40,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:40,094 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,094 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:40,094 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:40,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 04:53:40,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 04:53:40,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/hbase.id with ID: c5319186-f967-44a5-8ed9-cac9014de41a 2023-02-13 04:53:40,227 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:40,241 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:40,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1bf49b32 to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:40,319 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e22cb0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:40,341 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:40,343 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 04:53:40,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 04:53:40,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 04:53:40,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:40,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:40,365 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:40,396 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store-tmp 2023-02-13 04:53:40,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:40,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 04:53:40,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:40,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:40,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 04:53:40,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:40,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:40,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264020458Disabling compacts and flushes for region at 1676264020458Disabling writes for close at 1676264020459 (+1 ms)Writing region close event to WAL at 1676264020459Closed at 1676264020459 2023-02-13 04:53:40,460 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42441%2C1676264018749, suffix=, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/oldWALs, maxLogs=10 2023-02-13 04:53:40,547 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:40,547 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:40,547 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:40,555 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:40,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 2023-02-13 04:53:40,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK], DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK], DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK]] 2023-02-13 04:53:40,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:40,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:40,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:40,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:40,709 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:40,717 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 04:53:40,741 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 04:53:40,753 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:40,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:40,761 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:40,777 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:40,781 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:40,782 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64611683, jitterRate=-0.037210896611213684}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:40,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676264020666Initializing all the Stores at 1676264020669 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264020669Cleaning up temporary data from old regions at 1676264020768 (+99 ms)Cleaning up detritus from prior splits at 1676264020769 (+1 ms)Region opened successfully at 1676264020782 (+13 ms) 2023-02-13 04:53:40,783 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 04:53:40,805 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 04:53:40,805 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 04:53:40,808 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 04:53:40,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 04:53:40,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 39 msec 2023-02-13 04:53:40,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 04:53:40,878 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 04:53:40,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 04:53:40,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 04:53:40,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 04:53:40,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 04:53:40,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 04:53:40,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 04:53:40,923 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:40,925 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 04:53:40,925 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 04:53:40,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 04:53:40,947 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:40,947 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:40,947 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:40,947 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:40,947 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:40,950 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,42441,1676264018749, sessionid=0x1004c06f9cb0000, setting cluster-up flag (Was=false) 2023-02-13 04:53:40,964 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:40,971 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 04:53:40,974 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,980 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:40,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 04:53:40,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:40,993 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.hbase-snapshot/.tmp 2023-02-13 04:53:41,060 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(952): ClusterId : c5319186-f967-44a5-8ed9-cac9014de41a 2023-02-13 04:53:41,061 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(952): ClusterId : c5319186-f967-44a5-8ed9-cac9014de41a 2023-02-13 04:53:41,060 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(952): ClusterId : c5319186-f967-44a5-8ed9-cac9014de41a 2023-02-13 04:53:41,068 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:41,068 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:41,068 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:41,075 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:41,075 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:41,075 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:41,075 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:41,076 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:41,076 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:41,082 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:41,083 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:41,084 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:41,086 DEBUG [RS:2;jenkins-hbase4:37801] zookeeper.ReadOnlyZKClient(139): Connect 0x2c11e00b to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:41,086 DEBUG [RS:0;jenkins-hbase4:33925] zookeeper.ReadOnlyZKClient(139): Connect 0x31c23a8e to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:41,086 DEBUG [RS:1;jenkins-hbase4:36873] zookeeper.ReadOnlyZKClient(139): Connect 0x027dc5df to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:41,102 DEBUG [RS:2;jenkins-hbase4:37801] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22d61bf9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:41,103 DEBUG [RS:2;jenkins-hbase4:37801] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26e98b7b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:41,104 DEBUG [RS:1;jenkins-hbase4:36873] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cba2be0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:41,104 DEBUG [RS:1;jenkins-hbase4:36873] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@885aeaf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:41,108 DEBUG [RS:0;jenkins-hbase4:33925] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f05c2c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:41,108 DEBUG [RS:0;jenkins-hbase4:33925] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b3a51fe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:41,134 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:36873 2023-02-13 04:53:41,143 INFO [RS:1;jenkins-hbase4:36873] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:41,143 INFO [RS:1;jenkins-hbase4:36873] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:41,143 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:41,146 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42441,1676264018749 with isa=jenkins-hbase4.apache.org/172.31.14.131:36873, startcode=1676264020002 2023-02-13 04:53:41,136 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33925 2023-02-13 04:53:41,135 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:37801 2023-02-13 04:53:41,150 INFO [RS:0;jenkins-hbase4:33925] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:41,151 INFO [RS:0;jenkins-hbase4:33925] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:41,151 INFO [RS:2;jenkins-hbase4:37801] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:41,151 INFO [RS:2;jenkins-hbase4:37801] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:41,151 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:41,151 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:41,152 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42441,1676264018749 with isa=jenkins-hbase4.apache.org/172.31.14.131:37801, startcode=1676264020037 2023-02-13 04:53:41,153 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42441,1676264018749 with isa=jenkins-hbase4.apache.org/172.31.14.131:33925, startcode=1676264019948 2023-02-13 04:53:41,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:41,168 DEBUG [RS:2;jenkins-hbase4:37801] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:41,172 DEBUG [RS:1;jenkins-hbase4:36873] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:41,168 DEBUG [RS:0;jenkins-hbase4:33925] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:41,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676264051184 2023-02-13 04:53:41,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 04:53:41,190 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:41,191 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 04:53:41,198 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:41,202 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 04:53:41,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 04:53:41,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 04:53:41,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 04:53:41,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 04:53:41,220 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 04:53:41,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 04:53:41,233 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 04:53:41,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 04:53:41,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 04:53:41,239 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264021238,5,FailOnTimeoutGroup] 2023-02-13 04:53:41,239 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264021239,5,FailOnTimeoutGroup] 2023-02-13 04:53:41,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 04:53:41,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,243 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38325, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:41,243 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53887, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:41,243 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34711, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:41,262 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42441] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,262 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42441] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,286 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:41,287 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:41,287 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c 2023-02-13 04:53:41,290 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c 2023-02-13 04:53:41,290 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c 2023-02-13 04:53:41,290 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c 2023-02-13 04:53:41,291 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37257 2023-02-13 04:53:41,290 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37257 2023-02-13 04:53:41,291 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:41,291 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37257 2023-02-13 04:53:41,291 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:41,291 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:41,302 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:41,305 DEBUG [RS:0;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,306 WARN [RS:0;jenkins-hbase4:33925] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:41,306 DEBUG [RS:1;jenkins-hbase4:36873] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,306 INFO [RS:0;jenkins-hbase4:33925] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:41,306 DEBUG [RS:2;jenkins-hbase4:37801] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,306 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,306 WARN [RS:2;jenkins-hbase4:37801] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:41,306 WARN [RS:1;jenkins-hbase4:36873] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:41,307 INFO [RS:2;jenkins-hbase4:37801] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:41,307 INFO [RS:1;jenkins-hbase4:36873] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:41,307 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,307 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36873,1676264020002] 2023-02-13 04:53:41,307 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,308 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33925,1676264019948] 2023-02-13 04:53:41,308 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37801,1676264020037] 2023-02-13 04:53:41,321 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:41,323 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 04:53:41,324 DEBUG [RS:1;jenkins-hbase4:36873] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,324 DEBUG [RS:0;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,324 DEBUG [RS:2;jenkins-hbase4:37801] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,325 DEBUG [RS:1;jenkins-hbase4:36873] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,325 DEBUG [RS:0;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,325 DEBUG [RS:2;jenkins-hbase4:37801] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,326 DEBUG [RS:1;jenkins-hbase4:36873] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,326 DEBUG [RS:0;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,326 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/info 2023-02-13 04:53:41,326 DEBUG [RS:2;jenkins-hbase4:37801] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,327 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 04:53:41,328 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:41,328 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 04:53:41,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:41,331 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 04:53:41,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:41,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 04:53:41,335 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/table 2023-02-13 04:53:41,335 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 04:53:41,336 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:41,339 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740 2023-02-13 04:53:41,341 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:41,341 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:41,341 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740 2023-02-13 04:53:41,341 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:41,348 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 04:53:41,352 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 04:53:41,353 INFO [RS:0;jenkins-hbase4:33925] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:41,354 INFO [RS:1;jenkins-hbase4:36873] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:41,354 INFO [RS:2;jenkins-hbase4:37801] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:41,356 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:41,357 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72052388, jitterRate=0.07366424798965454}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 04:53:41,357 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676264021321Initializing all the Stores at 1676264021322 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264021323 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264021323Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264021323Cleaning up temporary data from old regions at 1676264021344 (+21 ms)Cleaning up detritus from prior splits at 1676264021345 (+1 ms)Region opened successfully at 1676264021357 (+12 ms) 2023-02-13 04:53:41,357 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 04:53:41,357 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 04:53:41,357 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 04:53:41,357 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 04:53:41,357 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 04:53:41,358 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:41,359 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264021357Disabling compacts and flushes for region at 1676264021357Disabling writes for close at 1676264021357Writing region close event to WAL at 1676264021358 (+1 ms)Closed at 1676264021358 2023-02-13 04:53:41,365 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:41,365 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 04:53:41,376 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 04:53:41,382 INFO [RS:1;jenkins-hbase4:36873] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:41,382 INFO [RS:2;jenkins-hbase4:37801] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:41,382 INFO [RS:0;jenkins-hbase4:33925] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:41,387 INFO [RS:0;jenkins-hbase4:33925] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:41,387 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,392 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 04:53:41,392 INFO [RS:2;jenkins-hbase4:37801] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:41,393 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:41,393 INFO [RS:1;jenkins-hbase4:36873] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:41,393 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,396 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,397 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:41,398 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 04:53:41,400 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:41,407 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,407 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,407 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,407 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,407 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,408 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,408 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,408 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:41,409 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:0;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:1;jenkins-hbase4:36873] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,409 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:41,410 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,410 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,411 DEBUG [RS:2;jenkins-hbase4:37801] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:41,413 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,413 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,414 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,430 INFO [RS:0;jenkins-hbase4:33925] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:41,430 INFO [RS:1;jenkins-hbase4:36873] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:41,431 INFO [RS:2;jenkins-hbase4:37801] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:41,434 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33925,1676264019948-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,434 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37801,1676264020037-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,434 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36873,1676264020002-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:41,455 INFO [RS:2;jenkins-hbase4:37801] regionserver.Replication(203): jenkins-hbase4.apache.org,37801,1676264020037 started 2023-02-13 04:53:41,455 INFO [RS:0;jenkins-hbase4:33925] regionserver.Replication(203): jenkins-hbase4.apache.org,33925,1676264019948 started 2023-02-13 04:53:41,455 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37801,1676264020037, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37801, sessionid=0x1004c06f9cb0003 2023-02-13 04:53:41,456 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33925,1676264019948, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33925, sessionid=0x1004c06f9cb0001 2023-02-13 04:53:41,456 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:41,456 DEBUG [RS:2;jenkins-hbase4:37801] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,456 DEBUG [RS:2;jenkins-hbase4:37801] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37801,1676264020037' 2023-02-13 04:53:41,456 DEBUG [RS:2;jenkins-hbase4:37801] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:41,457 DEBUG [RS:2;jenkins-hbase4:37801] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:41,457 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:41,457 DEBUG [RS:0;jenkins-hbase4:33925] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,458 DEBUG [RS:0;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33925,1676264019948' 2023-02-13 04:53:41,459 DEBUG [RS:0;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:41,459 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:41,459 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:41,459 DEBUG [RS:2;jenkins-hbase4:37801] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:41,459 INFO [RS:1;jenkins-hbase4:36873] regionserver.Replication(203): jenkins-hbase4.apache.org,36873,1676264020002 started 2023-02-13 04:53:41,459 DEBUG [RS:2;jenkins-hbase4:37801] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37801,1676264020037' 2023-02-13 04:53:41,459 DEBUG [RS:0;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:41,459 DEBUG [RS:2;jenkins-hbase4:37801] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:41,459 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36873,1676264020002, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36873, sessionid=0x1004c06f9cb0002 2023-02-13 04:53:41,459 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:41,459 DEBUG [RS:1;jenkins-hbase4:36873] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,459 DEBUG [RS:1;jenkins-hbase4:36873] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36873,1676264020002' 2023-02-13 04:53:41,459 DEBUG [RS:1;jenkins-hbase4:36873] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:41,460 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:41,460 DEBUG [RS:2;jenkins-hbase4:37801] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:41,460 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:41,460 DEBUG [RS:0;jenkins-hbase4:33925] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:41,460 DEBUG [RS:0;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33925,1676264019948' 2023-02-13 04:53:41,460 DEBUG [RS:0;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:41,460 DEBUG [RS:1;jenkins-hbase4:36873] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:41,461 DEBUG [RS:2;jenkins-hbase4:37801] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:41,461 INFO [RS:2;jenkins-hbase4:37801] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:41,461 DEBUG [RS:0;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:41,461 INFO [RS:2;jenkins-hbase4:37801] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:41,461 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:41,461 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:41,461 DEBUG [RS:1;jenkins-hbase4:36873] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,461 DEBUG [RS:1;jenkins-hbase4:36873] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36873,1676264020002' 2023-02-13 04:53:41,461 DEBUG [RS:1;jenkins-hbase4:36873] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:41,461 DEBUG [RS:0;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:41,462 INFO [RS:0;jenkins-hbase4:33925] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:41,462 INFO [RS:0;jenkins-hbase4:33925] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:41,463 DEBUG [RS:1;jenkins-hbase4:36873] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:41,463 DEBUG [RS:1;jenkins-hbase4:36873] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:41,463 INFO [RS:1;jenkins-hbase4:36873] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:41,463 INFO [RS:1;jenkins-hbase4:36873] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:41,552 DEBUG [jenkins-hbase4:42441] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 04:53:41,556 DEBUG [jenkins-hbase4:42441] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:41,562 DEBUG [jenkins-hbase4:42441] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:41,562 DEBUG [jenkins-hbase4:42441] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:41,562 DEBUG [jenkins-hbase4:42441] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:41,562 DEBUG [jenkins-hbase4:42441] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:41,565 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36873,1676264020002, state=OPENING 2023-02-13 04:53:41,572 DEBUG [PEWorker-5] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 04:53:41,573 INFO [RS:0;jenkins-hbase4:33925] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33925%2C1676264019948, suffix=, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs, maxLogs=32 2023-02-13 04:53:41,573 INFO [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36873%2C1676264020002, suffix=, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,36873,1676264020002, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs, maxLogs=32 2023-02-13 04:53:41,573 INFO [RS:2;jenkins-hbase4:37801] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37801%2C1676264020037, suffix=, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,37801,1676264020037, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs, maxLogs=32 2023-02-13 04:53:41,575 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:41,576 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 04:53:41,579 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36873,1676264020002}] 2023-02-13 04:53:41,604 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:41,604 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:41,616 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:41,630 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:41,630 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:41,631 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:41,631 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:41,632 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:41,632 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:41,649 INFO [RS:0;jenkins-hbase4:33925] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948/jenkins-hbase4.apache.org%2C33925%2C1676264019948.1676264021579 2023-02-13 04:53:41,650 DEBUG [RS:0;jenkins-hbase4:33925] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK], DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK], DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK]] 2023-02-13 04:53:41,651 INFO [RS:2;jenkins-hbase4:37801] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,37801,1676264020037/jenkins-hbase4.apache.org%2C37801%2C1676264020037.1676264021579 2023-02-13 04:53:41,651 DEBUG [RS:2;jenkins-hbase4:37801] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK], DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK], DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK]] 2023-02-13 04:53:41,652 INFO [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,36873,1676264020002/jenkins-hbase4.apache.org%2C36873%2C1676264020002.1676264021579 2023-02-13 04:53:41,652 DEBUG [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK], DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK], DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK]] 2023-02-13 04:53:41,766 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:41,768 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:41,772 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53894, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 04:53:41,787 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 04:53:41,787 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:41,791 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36873%2C1676264020002.meta, suffix=.meta, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,36873,1676264020002, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs, maxLogs=32 2023-02-13 04:53:41,810 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:41,810 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:41,812 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:41,820 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,36873,1676264020002/jenkins-hbase4.apache.org%2C36873%2C1676264020002.meta.1676264021792.meta 2023-02-13 04:53:41,821 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK], DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK], DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK]] 2023-02-13 04:53:41,821 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:41,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 04:53:41,841 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 04:53:41,846 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 04:53:41,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 04:53:41,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:41,853 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 04:53:41,853 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 04:53:41,856 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 04:53:41,859 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/info 2023-02-13 04:53:41,859 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/info 2023-02-13 04:53:41,860 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 04:53:41,861 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:41,861 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 04:53:41,863 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:41,863 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:41,864 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 04:53:41,864 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:41,865 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 04:53:41,866 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/table 2023-02-13 04:53:41,866 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/table 2023-02-13 04:53:41,867 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 04:53:41,867 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:41,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740 2023-02-13 04:53:41,872 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740 2023-02-13 04:53:41,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 04:53:41,877 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 04:53:41,880 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62358248, jitterRate=-0.07078969478607178}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 04:53:41,881 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676264021853Writing region info on filesystem at 1676264021853Initializing all the Stores at 1676264021855 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264021855Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264021855Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264021856 (+1 ms)Cleaning up temporary data from old regions at 1676264021873 (+17 ms)Cleaning up detritus from prior splits at 1676264021874 (+1 ms)Running coprocessor post-open hooks at 1676264021880 (+6 ms)Region opened successfully at 1676264021881 (+1 ms) 2023-02-13 04:53:41,891 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676264021758 2023-02-13 04:53:41,912 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 04:53:41,913 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 04:53:41,913 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36873,1676264020002, state=OPEN 2023-02-13 04:53:41,916 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 04:53:41,917 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 04:53:41,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 04:53:41,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36873,1676264020002 in 338 msec 2023-02-13 04:53:41,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 04:53:41,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 546 msec 2023-02-13 04:53:41,931 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 872 msec 2023-02-13 04:53:41,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676264020083, completionTime=-1 2023-02-13 04:53:41,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 04:53:41,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 04:53:42,031 DEBUG [hconnection-0x57c10ba2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:42,034 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53902, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:42,050 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 04:53:42,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676264082051 2023-02-13 04:53:42,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676264142051 2023-02-13 04:53:42,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 119 msec 2023-02-13 04:53:42,072 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42441,1676264018749-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:42,072 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42441,1676264018749-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:42,072 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42441,1676264018749-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:42,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42441, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:42,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:42,084 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 04:53:42,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 04:53:42,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:42,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 04:53:42,107 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:42,110 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:42,134 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,137 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc empty. 2023-02-13 04:53:42,138 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,138 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 04:53:42,174 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:42,176 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9afdb8e6bb1d6887dd3ec1d87c561bcc, NAME => 'hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp 2023-02-13 04:53:42,195 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:42,196 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 9afdb8e6bb1d6887dd3ec1d87c561bcc, disabling compactions & flushes 2023-02-13 04:53:42,196 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,196 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,196 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. after waiting 0 ms 2023-02-13 04:53:42,196 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,196 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,196 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 9afdb8e6bb1d6887dd3ec1d87c561bcc: Waiting for close lock at 1676264022195Disabling compacts and flushes for region at 1676264022195Disabling writes for close at 1676264022196 (+1 ms)Writing region close event to WAL at 1676264022196Closed at 1676264022196 2023-02-13 04:53:42,200 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:42,214 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264022203"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264022203"}]},"ts":"1676264022203"} 2023-02-13 04:53:42,241 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:42,243 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:42,247 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264022243"}]},"ts":"1676264022243"} 2023-02-13 04:53:42,250 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 04:53:42,255 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:42,256 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:42,256 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:42,256 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:42,256 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:42,259 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9afdb8e6bb1d6887dd3ec1d87c561bcc, ASSIGN}] 2023-02-13 04:53:42,262 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9afdb8e6bb1d6887dd3ec1d87c561bcc, ASSIGN 2023-02-13 04:53:42,263 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=9afdb8e6bb1d6887dd3ec1d87c561bcc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37801,1676264020037; forceNewPlan=false, retain=false 2023-02-13 04:53:42,415 INFO [jenkins-hbase4:42441] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:42,416 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9afdb8e6bb1d6887dd3ec1d87c561bcc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:42,416 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264022416"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264022416"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264022416"}]},"ts":"1676264022416"} 2023-02-13 04:53:42,420 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 9afdb8e6bb1d6887dd3ec1d87c561bcc, server=jenkins-hbase4.apache.org,37801,1676264020037}] 2023-02-13 04:53:42,574 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:42,574 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:42,579 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38916, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 04:53:42,585 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,587 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9afdb8e6bb1d6887dd3ec1d87c561bcc, NAME => 'hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:42,588 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,588 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:42,588 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,588 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,590 INFO [StoreOpener-9afdb8e6bb1d6887dd3ec1d87c561bcc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,592 DEBUG [StoreOpener-9afdb8e6bb1d6887dd3ec1d87c561bcc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/info 2023-02-13 04:53:42,592 DEBUG [StoreOpener-9afdb8e6bb1d6887dd3ec1d87c561bcc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/info 2023-02-13 04:53:42,593 INFO [StoreOpener-9afdb8e6bb1d6887dd3ec1d87c561bcc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9afdb8e6bb1d6887dd3ec1d87c561bcc columnFamilyName info 2023-02-13 04:53:42,594 INFO [StoreOpener-9afdb8e6bb1d6887dd3ec1d87c561bcc-1] regionserver.HStore(310): Store=9afdb8e6bb1d6887dd3ec1d87c561bcc/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:42,595 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,596 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,601 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:42,605 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:42,606 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9afdb8e6bb1d6887dd3ec1d87c561bcc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64066805, jitterRate=-0.04533021152019501}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:42,606 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9afdb8e6bb1d6887dd3ec1d87c561bcc: Running coprocessor pre-open hook at 1676264022588Writing region info on filesystem at 1676264022588Initializing all the Stores at 1676264022589 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264022590 (+1 ms)Cleaning up temporary data from old regions at 1676264022597 (+7 ms)Cleaning up detritus from prior splits at 1676264022598 (+1 ms)Running coprocessor post-open hooks at 1676264022606 (+8 ms)Region opened successfully at 1676264022606 2023-02-13 04:53:42,608 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc., pid=6, masterSystemTime=1676264022574 2023-02-13 04:53:42,613 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,613 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:42,615 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9afdb8e6bb1d6887dd3ec1d87c561bcc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:42,616 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264022614"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264022614"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264022614"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264022614"}]},"ts":"1676264022614"} 2023-02-13 04:53:42,626 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 04:53:42,626 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 9afdb8e6bb1d6887dd3ec1d87c561bcc, server=jenkins-hbase4.apache.org,37801,1676264020037 in 200 msec 2023-02-13 04:53:42,630 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 04:53:42,631 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=9afdb8e6bb1d6887dd3ec1d87c561bcc, ASSIGN in 367 msec 2023-02-13 04:53:42,632 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:42,632 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264022632"}]},"ts":"1676264022632"} 2023-02-13 04:53:42,635 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 04:53:42,638 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:42,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 543 msec 2023-02-13 04:53:42,707 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 04:53:42,709 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 04:53:42,709 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:42,731 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:42,732 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38920, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:42,745 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 04:53:42,761 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 04:53:42,766 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-02-13 04:53:42,779 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 04:53:42,792 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 04:53:42,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-02-13 04:53:42,805 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 04:53:42,809 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 04:53:42,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.714sec 2023-02-13 04:53:42,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 04:53:42,813 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 04:53:42,813 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 04:53:42,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42441,1676264018749-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 04:53:42,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42441,1676264018749-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 04:53:42,825 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 04:53:42,865 DEBUG [Listener at localhost/41311] zookeeper.ReadOnlyZKClient(139): Connect 0x763faadc to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:42,870 DEBUG [Listener at localhost/41311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3082d8e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:42,885 DEBUG [hconnection-0x3e88d58-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:42,896 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53916, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:42,908 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:42,915 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 04:53:42,915 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:42,916 INFO [Listener at localhost/41311] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 04:53:42,916 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 04:53:42,929 DEBUG [Listener at localhost/41311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 04:53:42,932 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36426, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 04:53:42,944 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:42,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 04:53:42,948 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:42,950 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:42,952 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 04:53:42,952 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:42,953 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a empty. 2023-02-13 04:53:42,955 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:42,955 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 04:53:42,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:42,977 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:42,978 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => f94e6da30a1d28bfa701f135a39ba73a, NAME => 'restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp 2023-02-13 04:53:42,994 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:42,994 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing f94e6da30a1d28bfa701f135a39ba73a, disabling compactions & flushes 2023-02-13 04:53:42,994 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:42,994 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:42,994 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. after waiting 0 ms 2023-02-13 04:53:42,994 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:42,994 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:42,994 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for f94e6da30a1d28bfa701f135a39ba73a: Waiting for close lock at 1676264022994Disabling compacts and flushes for region at 1676264022994Disabling writes for close at 1676264022994Writing region close event to WAL at 1676264022994Closed at 1676264022994 2023-02-13 04:53:42,998 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:43,000 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264023000"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264023000"}]},"ts":"1676264023000"} 2023-02-13 04:53:43,003 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:43,004 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:43,004 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264023004"}]},"ts":"1676264023004"} 2023-02-13 04:53:43,006 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 04:53:43,011 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:43,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:43,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:43,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:43,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:43,012 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f94e6da30a1d28bfa701f135a39ba73a, ASSIGN}] 2023-02-13 04:53:43,014 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f94e6da30a1d28bfa701f135a39ba73a, ASSIGN 2023-02-13 04:53:43,016 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=f94e6da30a1d28bfa701f135a39ba73a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37801,1676264020037; forceNewPlan=false, retain=false 2023-02-13 04:53:43,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:43,166 INFO [jenkins-hbase4:42441] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:43,167 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f94e6da30a1d28bfa701f135a39ba73a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:43,167 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264023167"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264023167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264023167"}]},"ts":"1676264023167"} 2023-02-13 04:53:43,170 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f94e6da30a1d28bfa701f135a39ba73a, server=jenkins-hbase4.apache.org,37801,1676264020037}] 2023-02-13 04:53:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:43,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:43,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f94e6da30a1d28bfa701f135a39ba73a, NAME => 'restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:43,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,331 INFO [StoreOpener-f94e6da30a1d28bfa701f135a39ba73a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,334 DEBUG [StoreOpener-f94e6da30a1d28bfa701f135a39ba73a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a/family 2023-02-13 04:53:43,334 DEBUG [StoreOpener-f94e6da30a1d28bfa701f135a39ba73a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a/family 2023-02-13 04:53:43,334 INFO [StoreOpener-f94e6da30a1d28bfa701f135a39ba73a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f94e6da30a1d28bfa701f135a39ba73a columnFamilyName family 2023-02-13 04:53:43,335 INFO [StoreOpener-f94e6da30a1d28bfa701f135a39ba73a-1] regionserver.HStore(310): Store=f94e6da30a1d28bfa701f135a39ba73a/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:43,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:43,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:43,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f94e6da30a1d28bfa701f135a39ba73a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59668449, jitterRate=-0.11087082326412201}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:43,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f94e6da30a1d28bfa701f135a39ba73a: Running coprocessor pre-open hook at 1676264023330Writing region info on filesystem at 1676264023330Initializing all the Stores at 1676264023331 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264023331Cleaning up temporary data from old regions at 1676264023338 (+7 ms)Cleaning up detritus from prior splits at 1676264023340 (+2 ms)Running coprocessor post-open hooks at 1676264023346 (+6 ms)Region opened successfully at 1676264023346 2023-02-13 04:53:43,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a., pid=11, masterSystemTime=1676264023323 2023-02-13 04:53:43,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:43,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:43,351 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f94e6da30a1d28bfa701f135a39ba73a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:43,352 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264023351"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264023351"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264023351"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264023351"}]},"ts":"1676264023351"} 2023-02-13 04:53:43,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 04:53:43,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f94e6da30a1d28bfa701f135a39ba73a, server=jenkins-hbase4.apache.org,37801,1676264020037 in 184 msec 2023-02-13 04:53:43,360 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 04:53:43,361 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=f94e6da30a1d28bfa701f135a39ba73a, ASSIGN in 345 msec 2023-02-13 04:53:43,362 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:43,362 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264023362"}]},"ts":"1676264023362"} 2023-02-13 04:53:43,364 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 04:53:43,367 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:43,369 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 423 msec 2023-02-13 04:53:43,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:43,572 INFO [Listener at localhost/41311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 04:53:43,572 DEBUG [Listener at localhost/41311] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 04:53:43,574 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:43,578 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 04:53:43,579 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:43,579 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 04:53:43,581 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:43,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 04:53:43,585 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:43,585 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 04:53:43,586 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:43,588 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,589 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c empty. 2023-02-13 04:53:43,590 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,590 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 04:53:43,616 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:43,617 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 27466f1fdc4879f7b76534c1c3c09a5c, NAME => 'restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp 2023-02-13 04:53:43,634 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:43,635 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 27466f1fdc4879f7b76534c1c3c09a5c, disabling compactions & flushes 2023-02-13 04:53:43,635 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,635 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,635 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. after waiting 0 ms 2023-02-13 04:53:43,635 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,635 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,635 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 27466f1fdc4879f7b76534c1c3c09a5c: Waiting for close lock at 1676264023635Disabling compacts and flushes for region at 1676264023635Disabling writes for close at 1676264023635Writing region close event to WAL at 1676264023635Closed at 1676264023635 2023-02-13 04:53:43,640 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:43,642 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264023642"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264023642"}]},"ts":"1676264023642"} 2023-02-13 04:53:43,645 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:43,646 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:43,647 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264023646"}]},"ts":"1676264023646"} 2023-02-13 04:53:43,649 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 04:53:43,653 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:43,654 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:43,654 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:43,655 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:43,655 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:43,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=27466f1fdc4879f7b76534c1c3c09a5c, ASSIGN}] 2023-02-13 04:53:43,657 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=27466f1fdc4879f7b76534c1c3c09a5c, ASSIGN 2023-02-13 04:53:43,659 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=27466f1fdc4879f7b76534c1c3c09a5c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37801,1676264020037; forceNewPlan=false, retain=false 2023-02-13 04:53:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:43,809 INFO [jenkins-hbase4:42441] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:43,810 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=27466f1fdc4879f7b76534c1c3c09a5c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:43,810 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264023810"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264023810"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264023810"}]},"ts":"1676264023810"} 2023-02-13 04:53:43,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 27466f1fdc4879f7b76534c1c3c09a5c, server=jenkins-hbase4.apache.org,37801,1676264020037}] 2023-02-13 04:53:43,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:43,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 27466f1fdc4879f7b76534c1c3c09a5c, NAME => 'restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:43,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:43,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,973 INFO [StoreOpener-27466f1fdc4879f7b76534c1c3c09a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,975 DEBUG [StoreOpener-27466f1fdc4879f7b76534c1c3c09a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c/family 2023-02-13 04:53:43,975 DEBUG [StoreOpener-27466f1fdc4879f7b76534c1c3c09a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c/family 2023-02-13 04:53:43,976 INFO [StoreOpener-27466f1fdc4879f7b76534c1c3c09a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 27466f1fdc4879f7b76534c1c3c09a5c columnFamilyName family 2023-02-13 04:53:43,977 INFO [StoreOpener-27466f1fdc4879f7b76534c1c3c09a5c-1] regionserver.HStore(310): Store=27466f1fdc4879f7b76534c1c3c09a5c/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:43,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:43,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:43,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 27466f1fdc4879f7b76534c1c3c09a5c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69157270, jitterRate=0.030523627996444702}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:43,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 27466f1fdc4879f7b76534c1c3c09a5c: Running coprocessor pre-open hook at 1676264023971Writing region info on filesystem at 1676264023971Initializing all the Stores at 1676264023973 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264023973Cleaning up temporary data from old regions at 1676264023982 (+9 ms)Cleaning up detritus from prior splits at 1676264023983 (+1 ms)Running coprocessor post-open hooks at 1676264023989 (+6 ms)Region opened successfully at 1676264023989 2023-02-13 04:53:43,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c., pid=14, masterSystemTime=1676264023966 2023-02-13 04:53:43,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:43,994 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=27466f1fdc4879f7b76534c1c3c09a5c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:43,994 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264023993"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264023993"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264023993"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264023993"}]},"ts":"1676264023993"} 2023-02-13 04:53:44,000 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 04:53:44,000 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 27466f1fdc4879f7b76534c1c3c09a5c, server=jenkins-hbase4.apache.org,37801,1676264020037 in 184 msec 2023-02-13 04:53:44,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 04:53:44,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=27466f1fdc4879f7b76534c1c3c09a5c, ASSIGN in 345 msec 2023-02-13 04:53:44,004 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:44,004 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264024004"}]},"ts":"1676264024004"} 2023-02-13 04:53:44,006 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 04:53:44,010 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:44,012 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 429 msec 2023-02-13 04:53:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:44,192 INFO [Listener at localhost/41311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 04:53:44,192 DEBUG [Listener at localhost/41311] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 04:53:44,192 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,196 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 04:53:44,196 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,196 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 04:53:44,198 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:44,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 04:53:44,201 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:44,202 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 04:53:44,203 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:44,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:44,205 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,206 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad empty. 2023-02-13 04:53:44,206 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,206 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 04:53:44,225 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:44,227 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => 44f7466753cf1e647dbfa6434db654ad, NAME => 'restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.tmp 2023-02-13 04:53:44,243 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:44,243 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing 44f7466753cf1e647dbfa6434db654ad, disabling compactions & flushes 2023-02-13 04:53:44,243 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,243 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,243 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. after waiting 0 ms 2023-02-13 04:53:44,243 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,243 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,243 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for 44f7466753cf1e647dbfa6434db654ad: Waiting for close lock at 1676264024243Disabling compacts and flushes for region at 1676264024243Disabling writes for close at 1676264024243Writing region close event to WAL at 1676264024243Closed at 1676264024243 2023-02-13 04:53:44,247 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:44,249 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264024249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264024249"}]},"ts":"1676264024249"} 2023-02-13 04:53:44,251 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:44,257 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:44,257 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264024257"}]},"ts":"1676264024257"} 2023-02-13 04:53:44,259 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 04:53:44,264 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:44,265 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:44,265 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:44,265 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:44,265 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:44,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=44f7466753cf1e647dbfa6434db654ad, ASSIGN}] 2023-02-13 04:53:44,267 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=44f7466753cf1e647dbfa6434db654ad, ASSIGN 2023-02-13 04:53:44,269 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=44f7466753cf1e647dbfa6434db654ad, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36873,1676264020002; forceNewPlan=false, retain=false 2023-02-13 04:53:44,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:44,419 INFO [jenkins-hbase4:42441] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:44,420 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=44f7466753cf1e647dbfa6434db654ad, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:44,420 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264024420"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264024420"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264024420"}]},"ts":"1676264024420"} 2023-02-13 04:53:44,423 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 44f7466753cf1e647dbfa6434db654ad, server=jenkins-hbase4.apache.org,36873,1676264020002}] 2023-02-13 04:53:44,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:44,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44f7466753cf1e647dbfa6434db654ad, NAME => 'restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:44,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:44,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,585 INFO [StoreOpener-44f7466753cf1e647dbfa6434db654ad-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,587 DEBUG [StoreOpener-44f7466753cf1e647dbfa6434db654ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad/family 2023-02-13 04:53:44,587 DEBUG [StoreOpener-44f7466753cf1e647dbfa6434db654ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad/family 2023-02-13 04:53:44,587 INFO [StoreOpener-44f7466753cf1e647dbfa6434db654ad-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f7466753cf1e647dbfa6434db654ad columnFamilyName family 2023-02-13 04:53:44,588 INFO [StoreOpener-44f7466753cf1e647dbfa6434db654ad-1] regionserver.HStore(310): Store=44f7466753cf1e647dbfa6434db654ad/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:44,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:44,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:44,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 44f7466753cf1e647dbfa6434db654ad; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65659370, jitterRate=-0.021599143743515015}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:44,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 44f7466753cf1e647dbfa6434db654ad: Running coprocessor pre-open hook at 1676264024583Writing region info on filesystem at 1676264024583Initializing all the Stores at 1676264024584 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264024584Cleaning up temporary data from old regions at 1676264024591 (+7 ms)Cleaning up detritus from prior splits at 1676264024592 (+1 ms)Running coprocessor post-open hooks at 1676264024601 (+9 ms)Region opened successfully at 1676264024601 2023-02-13 04:53:44,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad., pid=17, masterSystemTime=1676264024576 2023-02-13 04:53:44,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:44,605 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=44f7466753cf1e647dbfa6434db654ad, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:44,606 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264024605"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264024605"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264024605"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264024605"}]},"ts":"1676264024605"} 2023-02-13 04:53:44,611 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 04:53:44,611 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 44f7466753cf1e647dbfa6434db654ad, server=jenkins-hbase4.apache.org,36873,1676264020002 in 185 msec 2023-02-13 04:53:44,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 04:53:44,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=44f7466753cf1e647dbfa6434db654ad, ASSIGN in 346 msec 2023-02-13 04:53:44,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:44,616 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264024615"}]},"ts":"1676264024615"} 2023-02-13 04:53:44,617 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 04:53:44,620 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:44,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 422 msec 2023-02-13 04:53:44,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42441] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:44,808 INFO [Listener at localhost/41311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 04:53:44,808 DEBUG [Listener at localhost/41311] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 04:53:44,809 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,813 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 04:53:44,813 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,813 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 04:53:44,813 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,820 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,823 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,827 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:44,829 INFO [Listener at localhost/41311] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 04:53:44,838 INFO [Listener at localhost/41311] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 04:53:44,839 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(157): Stopping HMaster and jenkins-hbase4.apache.org,33925,1676264019948 server 2023-02-13 04:53:44,839 INFO [Listener at localhost/41311] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:42441,5,FailOnTimeoutGroup] 2023-02-13 04:53:44,839 INFO [Listener at localhost/41311] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42441,1676264018749' ***** 2023-02-13 04:53:44,839 INFO [Listener at localhost/41311] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 04:53:44,840 DEBUG [M:0;jenkins-hbase4:42441] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22ef996d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:44,840 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:44,845 DEBUG [M:0;jenkins-hbase4:42441] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1bf49b32 to 127.0.0.1:52493 2023-02-13 04:53:44,846 DEBUG [M:0;jenkins-hbase4:42441] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:44,846 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42441,1676264018749; all regions closed. 2023-02-13 04:53:44,846 DEBUG [M:0;jenkins-hbase4:42441] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:44,847 DEBUG [M:0;jenkins-hbase4:42441] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 04:53:44,847 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 04:53:44,847 DEBUG [M:0;jenkins-hbase4:42441] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 04:53:44,847 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264021239] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264021239,5,FailOnTimeoutGroup] 2023-02-13 04:53:44,847 INFO [M:0;jenkins-hbase4:42441] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 04:53:44,847 INFO [M:0;jenkins-hbase4:42441] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 04:53:44,847 INFO [M:0;jenkins-hbase4:42441] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 04:53:44,848 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264021238] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264021238,5,FailOnTimeoutGroup] 2023-02-13 04:53:44,848 DEBUG [M:0;jenkins-hbase4:42441] master.HMaster(1502): Stopping service threads 2023-02-13 04:53:44,849 INFO [M:0;jenkins-hbase4:42441] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 04:53:44,849 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:44,849 INFO [M:0;jenkins-hbase4:42441] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 04:53:44,849 INFO [Listener at localhost/41311] hbase.LocalHBaseCluster(360): Waiting on jenkins-hbase4.apache.org,42441,1676264018749 2023-02-13 04:53:44,849 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:44,849 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:44,851 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:44,850 ERROR [M:0;jenkins-hbase4:42441] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-02-13 04:53:44,851 INFO [M:0;jenkins-hbase4:42441] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 04:53:44,851 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 04:53:44,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:44,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:44,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:44,854 DEBUG [M:0;jenkins-hbase4:42441] zookeeper.ZKUtil(398): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 04:53:44,855 WARN [M:0;jenkins-hbase4:42441] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 04:53:44,855 INFO [M:0;jenkins-hbase4:42441] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 04:53:44,855 INFO [M:0;jenkins-hbase4:42441] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 04:53:44,856 DEBUG [M:0;jenkins-hbase4:42441] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 04:53:44,856 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:44,856 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:44,856 DEBUG [M:0;jenkins-hbase4:42441] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:44,856 DEBUG [M:0;jenkins-hbase4:42441] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 04:53:44,856 DEBUG [M:0;jenkins-hbase4:42441] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:44,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:44,857 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.23 KB heapSize=76.14 KB 2023-02-13 04:53:44,951 INFO [M:0;jenkins-hbase4:42441] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.23 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d048486f332f42c5abbb9503fba22d4c 2023-02-13 04:53:44,992 DEBUG [M:0;jenkins-hbase4:42441] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d048486f332f42c5abbb9503fba22d4c as hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d048486f332f42c5abbb9503fba22d4c 2023-02-13 04:53:45,000 INFO [M:0;jenkins-hbase4:42441] regionserver.HStore(1080): Added hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d048486f332f42c5abbb9503fba22d4c, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 04:53:45,002 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegion(2947): Finished flush of dataSize ~64.23 KB/65767, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 145ms, sequenceid=141, compaction requested=false 2023-02-13 04:53:45,006 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:45,007 DEBUG [M:0;jenkins-hbase4:42441] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264024855Disabling compacts and flushes for region at 1676264024855Disabling writes for close at 1676264024856 (+1 ms)Obtaining lock to block concurrent updates at 1676264024857 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676264024857Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65767, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676264024866 (+9 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676264024867 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676264024870 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676264024906 (+36 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676264024911 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676264024993 (+82 ms)Finished flush of dataSize ~64.23 KB/65767, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 145ms, sequenceid=141, compaction requested=false at 1676264025002 (+9 ms)Writing region close event to WAL at 1676264025006 (+4 ms)Closed at 1676264025006 2023-02-13 04:53:45,014 INFO [M:0;jenkins-hbase4:42441] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 04:53:45,014 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:45,015 INFO [M:0;jenkins-hbase4:42441] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42441 2023-02-13 04:53:45,023 DEBUG [M:0;jenkins-hbase4:42441] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42441,1676264018749 already deleted, retry=false 2023-02-13 04:53:45,030 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42441 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:45,031 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42441 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 2023-02-13 04:53:45,032 DEBUG [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:45,038 WARN [RS-EventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42441 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:45,039 DEBUG [RS-EventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42441 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 2023-02-13 04:53:45,040 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:45,086 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42441 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:45,086 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42441 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 2023-02-13 04:53:45,087 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:45,126 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:45,126 INFO [M:0;jenkins-hbase4:42441] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42441,1676264018749; zookeeper connection closed. 2023-02-13 04:53:45,127 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ab70e0d rejected from java.util.concurrent.ThreadPoolExecutor@54f15a6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:45,127 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:42441-0x1004c06f9cb0000, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:45,127 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@538c6d89 rejected from java.util.concurrent.ThreadPoolExecutor@54f15a6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:45,128 INFO [Listener at localhost/41311] hbase.MiniHBaseCluster(508): Stopping Thread[RS:0;jenkins-hbase4:33925,5,FailOnTimeoutGroup] 2023-02-13 04:53:45,129 INFO [Listener at localhost/41311] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33925,1676264019948' ***** 2023-02-13 04:53:45,129 INFO [Listener at localhost/41311] regionserver.HRegionServer(2310): STOPPED: Stopping rs 0 2023-02-13 04:53:45,129 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(163): Sleeping a bit 2023-02-13 04:53:45,219 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:45,233 INFO [RS:0;jenkins-hbase4:33925] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:45,233 INFO [RS:0;jenkins-hbase4:33925] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:45,233 INFO [RS:0;jenkins-hbase4:33925] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:45,233 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:45,233 DEBUG [RS:0;jenkins-hbase4:33925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31c23a8e to 127.0.0.1:52493 2023-02-13 04:53:45,233 DEBUG [RS:0;jenkins-hbase4:33925] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:45,233 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33925,1676264019948; all regions closed. 2023-02-13 04:53:45,246 DEBUG [RS:0;jenkins-hbase4:33925] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs 2023-02-13 04:53:45,247 INFO [RS:0;jenkins-hbase4:33925] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33925%2C1676264019948:(num 1676264021579) 2023-02-13 04:53:45,247 DEBUG [RS:0;jenkins-hbase4:33925] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:45,247 INFO [RS:0;jenkins-hbase4:33925] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:45,247 INFO [RS:0;jenkins-hbase4:33925] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:45,247 INFO [RS:0;jenkins-hbase4:33925] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:45,247 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:45,247 INFO [RS:0;jenkins-hbase4:33925] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:45,247 INFO [RS:0;jenkins-hbase4:33925] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:45,248 INFO [RS:0;jenkins-hbase4:33925] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33925 2023-02-13 04:53:45,251 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:45,251 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:45,251 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 2023-02-13 04:53:45,251 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:45,251 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:45,251 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:45,251 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6bcd7ffd rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:45,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:45,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:45,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:45,254 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 znode expired, triggering replicatorRemoved event 2023-02-13 04:53:45,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:45,255 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33925,1676264019948 znode expired, triggering replicatorRemoved event 2023-02-13 04:53:45,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:45,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:45,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:45,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:45,318 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:45,440 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:45,440 INFO [RS:0;jenkins-hbase4:33925] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33925,1676264019948; zookeeper connection closed. 2023-02-13 04:53:45,440 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ba2ee59 rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:45,440 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb0001, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:45,440 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63a89d7f rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:45,440 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7ac1d1fa] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7ac1d1fa 2023-02-13 04:53:46,044 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:46,090 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:47,048 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:47,093 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 04:53:47,129 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(166): Starting HMaster and region server jenkins-hbase4.apache.org,33925,1676264019948 second time with the same port 2023-02-13 04:53:47,143 INFO [Listener at localhost/41311] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:47,144 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:47,144 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:47,144 INFO [Listener at localhost/41311] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:47,144 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:47,144 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:47,145 INFO [Listener at localhost/41311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:47,148 INFO [Listener at localhost/41311] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36759 2023-02-13 04:53:47,149 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:47,151 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:47,152 INFO [Listener at localhost/41311] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36759 connecting to ZooKeeper ensemble=127.0.0.1:52493 2023-02-13 04:53:47,158 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:367590x0, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:47,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36759-0x1004c06f9cb0009 connected 2023-02-13 04:53:47,163 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:47,164 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(162): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 04:53:47,165 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:47,166 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36759 2023-02-13 04:53:47,166 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36759 2023-02-13 04:53:47,166 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36759 2023-02-13 04:53:47,166 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36759 2023-02-13 04:53:47,167 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36759 2023-02-13 04:53:47,167 INFO [Listener at localhost/41311] master.HMaster(439): hbase.rootdir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c, hbase.cluster.distributed=false 2023-02-13 04:53:47,168 DEBUG [pool-218-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 04:53:47,171 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,172 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:47,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,174 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:47,174 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:47,175 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:47,175 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:47,175 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:47,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,36759,1676264027143 from backup master directory 2023-02-13 04:53:47,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:47,177 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegionServer(952): ClusterId : c5319186-f967-44a5-8ed9-cac9014de41a 2023-02-13 04:53:47,178 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,178 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:47,178 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:47,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] client.ConnectionUtils(127): regionserver/jenkins-hbase4:33925 server-side Connection retries=45 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:47,184 INFO [Listener at localhost/41311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:47,186 INFO [Listener at localhost/41311] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33925 2023-02-13 04:53:47,186 INFO [Listener at localhost/41311] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:47,188 DEBUG [Listener at localhost/41311] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:47,189 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:47,190 INFO [Listener at localhost/41311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:47,191 INFO [Listener at localhost/41311] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33925 connecting to ZooKeeper ensemble=127.0.0.1:52493 2023-02-13 04:53:47,199 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:339250x0, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:47,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33925-0x1004c06f9cb000a connected 2023-02-13 04:53:47,203 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:47,204 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 04:53:47,206 DEBUG [Listener at localhost/41311] zookeeper.ZKUtil(164): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:47,208 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33925 2023-02-13 04:53:47,208 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33925 2023-02-13 04:53:47,209 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33925 2023-02-13 04:53:47,209 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33925 2023-02-13 04:53:47,209 DEBUG [Listener at localhost/41311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33925 2023-02-13 04:53:47,212 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(952): ClusterId : c5319186-f967-44a5-8ed9-cac9014de41a 2023-02-13 04:53:47,212 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:47,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:47,215 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:47,215 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:47,217 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:47,218 DEBUG [RS:3;jenkins-hbase4:33925] zookeeper.ReadOnlyZKClient(139): Connect 0x5a78b3c7 to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:47,224 DEBUG [RS:3;jenkins-hbase4:33925] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54bf56f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:47,225 DEBUG [RS:3;jenkins-hbase4:33925] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5080d8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:47,239 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:33925 2023-02-13 04:53:47,239 INFO [RS:3;jenkins-hbase4:33925] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:47,239 INFO [RS:3;jenkins-hbase4:33925] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:47,239 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:47,240 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36759,1676264027143 with isa=jenkins-hbase4.apache.org/172.31.14.131:33925, startcode=1676264027183 2023-02-13 04:53:47,241 DEBUG [RS:3;jenkins-hbase4:33925] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:47,246 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59383, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:47,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36759] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:47,259 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x559b56ce to 127.0.0.1:52493 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:47,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5238150c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:47,263 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:47,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 04:53:47,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:47,271 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 04:53:47,271 WARN [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 04:53:47,274 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749 to hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749-dead as it is dead 2023-02-13 04:53:47,276 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749-dead/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 2023-02-13 04:53:47,280 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749-dead/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 after 4ms 2023-02-13 04:53:47,282 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749-dead/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 to hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 2023-02-13 04:53:47,282 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,42441,1676264018749-dead 2023-02-13 04:53:47,283 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36759%2C1676264027143, suffix=, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,36759,1676264027143, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/oldWALs, maxLogs=10 2023-02-13 04:53:47,301 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:47,303 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:47,303 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:47,305 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/WALs/jenkins-hbase4.apache.org,36759,1676264027143/jenkins-hbase4.apache.org%2C36759%2C1676264027143.1676264027286 2023-02-13 04:53:47,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK], DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK], DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK]] 2023-02-13 04:53:47,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:47,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:47,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:47,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:47,308 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:47,310 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 04:53:47,310 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 04:53:47,318 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d048486f332f42c5abbb9503fba22d4c 2023-02-13 04:53:47,319 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:47,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 04:53:47,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 2023-02-13 04:53:47,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 2023-02-13 04:53:47,345 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42441%2C1676264018749.1676264020496 2023-02-13 04:53:47,349 DEBUG [RS:2;jenkins-hbase4:37801] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:47,350 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:47,352 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52163, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:47,353 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36759] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:47,354 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 04:53:47,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64462014, jitterRate=-0.03944113850593567}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:47,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676264027306Initializing all the Stores at 1676264027307 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264027307Cleaning up temporary data from old regions at 1676264027346 (+39 ms)Cleaning up detritus from prior splits at 1676264027347 (+1 ms)Region opened successfully at 1676264027355 (+8 ms) 2023-02-13 04:53:47,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 04:53:47,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 04:53:47,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 04:53:47,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 04:53:47,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 04:53:47,372 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:47,372 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36759,1676264027143 with isa=jenkins-hbase4.apache.org/172.31.14.131:33925, startcode=1676264027183 2023-02-13 04:53:47,373 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 04:53:47,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36759] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:47,373 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 04:53:47,373 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 04:53:47,374 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 04:53:47,374 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 04:53:47,374 WARN [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 04:53:47,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 04:53:47,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 04:53:47,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 04:53:47,377 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 19 msec 2023-02-13 04:53:47,377 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 04:53:47,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 04:53:47,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,36873,1676264020002, table=hbase:meta, region=1588230740 2023-02-13 04:53:47,381 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 04:53:47,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:47,386 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,389 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33925,1676264019948 already deleted, retry=false 2023-02-13 04:53:47,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,33925,1676264019948 on jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,395 DEBUG [RS:1;jenkins-hbase4:36873] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:47,396 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36109, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:47,397 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36759] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:47,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,33925,1676264019948, splitWal=true, meta=false 2023-02-13 04:53:47,401 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,33925,1676264019948 (carryingMeta=false) jenkins-hbase4.apache.org,33925,1676264019948/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@13f52ced[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 04:53:47,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 04:53:47,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 04:53:47,419 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 04:53:47,419 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 04:53:47,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 04:53:47,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 04:53:47,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 04:53:47,422 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 04:53:47,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,36759,1676264027143, sessionid=0x1004c06f9cb0009, setting cluster-up flag (Was=true) 2023-02-13 04:53:47,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 04:53:47,425 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 04:53:47,429 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,430 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/.hbase-snapshot/.tmp 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:47,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676264057440 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 04:53:47,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 04:53:47,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 04:53:47,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 04:53:47,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 04:53:47,443 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33925,1676264019948; numProcessing=1 2023-02-13 04:53:47,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 04:53:47,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264027444,5,FailOnTimeoutGroup] 2023-02-13 04:53:47,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264027444,5,FailOnTimeoutGroup] 2023-02-13 04:53:47,444 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33925,1676264019948, splitWal=true, meta=false 2023-02-13 04:53:47,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 04:53:47,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676264027173, completionTime=-1 2023-02-13 04:53:47,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 04:53:47,454 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 04:53:47,455 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableThree' 2023-02-13 04:53:47,457 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 04:53:47,458 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableTwo' 2023-02-13 04:53:47,459 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableOne' 2023-02-13 04:53:47,575 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36759,1676264027143 with isa=jenkins-hbase4.apache.org/172.31.14.131:33925, startcode=1676264027183 2023-02-13 04:53:47,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36759] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,577 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c 2023-02-13 04:53:47,577 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37257 2023-02-13 04:53:47,577 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:47,579 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,579 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,579 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,580 DEBUG [RS:3;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,580 WARN [RS:3;jenkins-hbase4:33925] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:47,580 INFO [RS:3;jenkins-hbase4:33925] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:47,580 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:47,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:47,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,583 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33925,1676264027183] 2023-02-13 04:53:47,586 DEBUG [RS:3;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:47,587 DEBUG [RS:3;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,587 DEBUG [RS:3;jenkins-hbase4:33925] zookeeper.ZKUtil(162): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,588 DEBUG [RS:3;jenkins-hbase4:33925] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:47,589 INFO [RS:3;jenkins-hbase4:33925] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:47,591 INFO [RS:3;jenkins-hbase4:33925] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:47,592 INFO [RS:3;jenkins-hbase4:33925] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:47,592 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,592 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:47,594 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,594 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,594 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,594 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,595 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=151ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,595 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:33925, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,595 DEBUG [RS:3;jenkins-hbase4:33925] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:33925, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:47,604 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,604 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,604 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,605 DEBUG [hconnection-0x3454b7d0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:47,613 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 04:53:47,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=9afdb8e6bb1d6887dd3ec1d87c561bcc, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,37801,1676264020037, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037, openSeqNum=2 2023-02-13 04:53:47,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=f94e6da30a1d28bfa701f135a39ba73a, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,37801,1676264020037, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037, openSeqNum=2 2023-02-13 04:53:47,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=44f7466753cf1e647dbfa6434db654ad, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36873,1676264020002, regionLocation=jenkins-hbase4.apache.org,36873,1676264020002, openSeqNum=2 2023-02-13 04:53:47,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=27466f1fdc4879f7b76534c1c3c09a5c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,37801,1676264020037, regionLocation=jenkins-hbase4.apache.org,37801,1676264020037, openSeqNum=2 2023-02-13 04:53:47,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 04:53:47,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676264087618 2023-02-13 04:53:47,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676264147618 2023-02-13 04:53:47,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 22 msec 2023-02-13 04:53:47,620 INFO [RS:3;jenkins-hbase4:33925] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:47,620 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33925,1676264027183-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,639 INFO [RS:3;jenkins-hbase4:33925] regionserver.Replication(203): jenkins-hbase4.apache.org,33925,1676264027183 started 2023-02-13 04:53:47,639 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33925,1676264027183, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33925, sessionid=0x1004c06f9cb000a 2023-02-13 04:53:47,639 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:47,639 DEBUG [RS:3;jenkins-hbase4:33925] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,639 DEBUG [RS:3;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33925,1676264027183' 2023-02-13 04:53:47,639 DEBUG [RS:3;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:47,640 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:47,641 DEBUG [RS:3;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:47,645 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:47,646 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:47,646 DEBUG [RS:3;jenkins-hbase4:33925] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,646 DEBUG [RS:3;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33925,1676264027183' 2023-02-13 04:53:47,646 WARN [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42441 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:47,646 DEBUG [RS:3;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:47,646 WARN [Listener at localhost/41311] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:42441 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:47,646 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42441 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42441 2023-02-13 04:53:47,648 DEBUG [Listener at localhost/41311] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42441 this server is in the failed servers list 2023-02-13 04:53:47,648 DEBUG [RS:3;jenkins-hbase4:33925] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:47,648 WARN [Listener at localhost/41311] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:42441 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:42441 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnSingleRSRestart(TestRetainAssignmentOnRestart.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:42441 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 04:53:47,648 DEBUG [RS:3;jenkins-hbase4:33925] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:47,649 INFO [RS:3;jenkins-hbase4:33925] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:47,649 INFO [RS:3;jenkins-hbase4:33925] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:47,650 DEBUG [Listener at localhost/41311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 04:53:47,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36759,1676264027143-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36759,1676264027143-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36759,1676264027143-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:36759, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:47,653 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,33925,1676264019948 had 0 regions 2023-02-13 04:53:47,653 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 04:53:47,661 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33698, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 04:53:47,661 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33925,1676264019948, splitWal=true, meta=false, isMeta: false 2023-02-13 04:53:47,664 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948-splitting 2023-02-13 04:53:47,667 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 04:53:47,669 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948-splitting dir is empty, no logs to split. 2023-02-13 04:53:47,669 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,33925,1676264019948 WAL count=0, meta=false 2023-02-13 04:53:47,675 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948-splitting dir is empty, no logs to split. 2023-02-13 04:53:47,675 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,33925,1676264019948 WAL count=0, meta=false 2023-02-13 04:53:47,675 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33925,1676264019948 WAL splitting is done? wals=0, meta=false 2023-02-13 04:53:47,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36759] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36759] ipc.CallRunner(144): callId: 48 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:33698 deadline: 1676264087677, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 04:53:47,679 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 04:53:47,679 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,33925,1676264019948 failed, ignore...File hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264019948-splitting does not exist. 2023-02-13 04:53:47,682 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,33925,1676264019948 after splitting done 2023-02-13 04:53:47,682 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,33925,1676264019948 from processing; numProcessing=0 2023-02-13 04:53:47,687 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,33925,1676264019948, splitWal=true, meta=false in 290 msec 2023-02-13 04:53:47,688 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 04:53:47,692 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 04:53:47,692 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 0.514sec 2023-02-13 04:53:47,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 04:53:47,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 04:53:47,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 04:53:47,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36759,1676264027143-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 04:53:47,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36759,1676264027143-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 04:53:47,701 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 04:53:47,752 INFO [RS:3;jenkins-hbase4:33925] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33925%2C1676264027183, suffix=, logDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264027183, archiveDir=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs, maxLogs=32 2023-02-13 04:53:47,776 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK] 2023-02-13 04:53:47,776 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK] 2023-02-13 04:53:47,776 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK] 2023-02-13 04:53:47,781 INFO [RS:3;jenkins-hbase4:33925] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/WALs/jenkins-hbase4.apache.org,33925,1676264027183/jenkins-hbase4.apache.org%2C33925%2C1676264027183.1676264027753 2023-02-13 04:53:47,781 DEBUG [RS:3;jenkins-hbase4:33925] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36071,DS-06b0245c-348a-46fe-a73a-e9588f517ee5,DISK], DatanodeInfoWithStorage[127.0.0.1:35517,DS-7f2ebf67-572c-41a2-8f0f-b4459dc7c10c,DISK], DatanodeInfoWithStorage[127.0.0.1:35683,DS-40d161fd-6e75-44c9-bccd-3a1cbf486688,DISK]] 2023-02-13 04:53:47,794 DEBUG [hconnection-0x3e88d58-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:47,798 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38908, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:47,804 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:47,824 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:47,834 INFO [Listener at localhost/41311] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:47,834 INFO [Listener at localhost/41311] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 04:53:47,837 INFO [Listener at localhost/41311] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 04:53:47,837 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 9afdb8e6bb1d6887dd3ec1d87c561bcc, NAME => 'hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,37801,1676264020037, currentServer=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,838 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => f94e6da30a1d28bfa701f135a39ba73a, NAME => 'restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,37801,1676264020037, currentServer=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,838 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 44f7466753cf1e647dbfa6434db654ad, NAME => 'restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,36873,1676264020002, currentServer=jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:47,838 INFO [Listener at localhost/41311] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 27466f1fdc4879f7b76534c1c3c09a5c, NAME => 'restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,37801,1676264020037, currentServer=jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,838 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 04:53:47,838 INFO [Listener at localhost/41311] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 04:53:47,841 DEBUG [Listener at localhost/41311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x763faadc to 127.0.0.1:52493 2023-02-13 04:53:47,841 DEBUG [Listener at localhost/41311] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:47,841 DEBUG [Listener at localhost/41311] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 04:53:47,841 DEBUG [Listener at localhost/41311] util.JVMClusterUtil(257): Found active master hash=1933868872, stopped=false 2023-02-13 04:53:47,842 INFO [Listener at localhost/41311] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:47,844 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:47,844 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:47,844 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:47,844 INFO [Listener at localhost/41311] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 04:53:47,844 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:47,844 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:47,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:47,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:47,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:47,846 DEBUG [Listener at localhost/41311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x559b56ce to 127.0.0.1:52493 2023-02-13 04:53:47,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:47,846 DEBUG [Listener at localhost/41311] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:47,846 INFO [Listener at localhost/41311] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36873,1676264020002' ***** 2023-02-13 04:53:47,847 INFO [Listener at localhost/41311] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:47,847 INFO [Listener at localhost/41311] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37801,1676264020037' ***** 2023-02-13 04:53:47,847 INFO [Listener at localhost/41311] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:47,847 INFO [RS:1;jenkins-hbase4:36873] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:47,847 INFO [RS:2;jenkins-hbase4:37801] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:47,847 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:47,847 INFO [Listener at localhost/41311] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33925,1676264027183' ***** 2023-02-13 04:53:47,847 INFO [RS:2;jenkins-hbase4:37801] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:47,847 INFO [Listener at localhost/41311] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:47,847 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:47,847 INFO [RS:1;jenkins-hbase4:36873] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:47,847 INFO [RS:3;jenkins-hbase4:33925] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:47,847 INFO [RS:2;jenkins-hbase4:37801] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:47,847 INFO [RS:1;jenkins-hbase4:36873] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:47,848 INFO [RS:3;jenkins-hbase4:33925] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:47,848 INFO [RS:3;jenkins-hbase4:33925] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:47,848 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:47,848 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,848 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(3304): Received CLOSE for 27466f1fdc4879f7b76534c1c3c09a5c 2023-02-13 04:53:47,848 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(3304): Received CLOSE for 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:47,849 DEBUG [RS:3;jenkins-hbase4:33925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a78b3c7 to 127.0.0.1:52493 2023-02-13 04:53:47,849 DEBUG [RS:3;jenkins-hbase4:33925] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:47,849 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33925,1676264027183; all regions closed. 2023-02-13 04:53:47,850 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(3304): Received CLOSE for 9afdb8e6bb1d6887dd3ec1d87c561bcc 2023-02-13 04:53:47,850 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(3304): Received CLOSE for f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:47,850 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:47,850 DEBUG [RS:2;jenkins-hbase4:37801] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c11e00b to 127.0.0.1:52493 2023-02-13 04:53:47,850 DEBUG [RS:2;jenkins-hbase4:37801] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:47,850 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 04:53:47,850 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1479): Online Regions={27466f1fdc4879f7b76534c1c3c09a5c=restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c., 9afdb8e6bb1d6887dd3ec1d87c561bcc=hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc., f94e6da30a1d28bfa701f135a39ba73a=restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a.} 2023-02-13 04:53:47,851 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:47,851 DEBUG [RS:1;jenkins-hbase4:36873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x027dc5df to 127.0.0.1:52493 2023-02-13 04:53:47,851 DEBUG [RS:1;jenkins-hbase4:36873] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:47,851 INFO [RS:1;jenkins-hbase4:36873] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:47,851 INFO [RS:1;jenkins-hbase4:36873] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:47,851 INFO [RS:1;jenkins-hbase4:36873] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:47,851 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 04:53:47,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 27466f1fdc4879f7b76534c1c3c09a5c, disabling compactions & flushes 2023-02-13 04:53:47,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 44f7466753cf1e647dbfa6434db654ad, disabling compactions & flushes 2023-02-13 04:53:47,853 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 04:53:47,853 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1479): Online Regions={44f7466753cf1e647dbfa6434db654ad=restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 04:53:47,854 DEBUG [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1505): Waiting on 27466f1fdc4879f7b76534c1c3c09a5c, 9afdb8e6bb1d6887dd3ec1d87c561bcc, f94e6da30a1d28bfa701f135a39ba73a 2023-02-13 04:53:47,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:47,854 DEBUG [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1505): Waiting on 1588230740, 44f7466753cf1e647dbfa6434db654ad 2023-02-13 04:53:47,853 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 04:53:47,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:47,854 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 04:53:47,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:47,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. after waiting 0 ms 2023-02-13 04:53:47,854 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 04:53:47,857 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 04:53:47,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:47,857 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 04:53:47,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:47,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. after waiting 0 ms 2023-02-13 04:53:47,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:47,857 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 04:53:47,862 DEBUG [RS:3;jenkins-hbase4:33925] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs 2023-02-13 04:53:47,862 INFO [RS:3;jenkins-hbase4:33925] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33925%2C1676264027183:(num 1676264027753) 2023-02-13 04:53:47,863 DEBUG [RS:3;jenkins-hbase4:33925] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:47,863 INFO [RS:3;jenkins-hbase4:33925] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:47,863 INFO [RS:3;jenkins-hbase4:33925] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:33925 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:47,863 INFO [RS:3;jenkins-hbase4:33925] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:47,863 INFO [regionserver/jenkins-hbase4:33925.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:47,863 INFO [RS:3;jenkins-hbase4:33925] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:47,863 INFO [RS:3;jenkins-hbase4:33925] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:47,864 INFO [RS:3;jenkins-hbase4:33925] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33925 2023-02-13 04:53:47,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableTwo/27466f1fdc4879f7b76534c1c3c09a5c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 04:53:47,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableThree/44f7466753cf1e647dbfa6434db654ad/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 04:53:47,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:47,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:47,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 44f7466753cf1e647dbfa6434db654ad: Waiting for close lock at 1676264027851Running coprocessor pre-close hooks at 1676264027851Disabling compacts and flushes for region at 1676264027851Disabling writes for close at 1676264027857 (+6 ms)Writing region close event to WAL at 1676264027858 (+1 ms)Running coprocessor post-close hooks at 1676264027868 (+10 ms)Closed at 1676264027869 (+1 ms) 2023-02-13 04:53:47,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 27466f1fdc4879f7b76534c1c3c09a5c: Waiting for close lock at 1676264027850Running coprocessor pre-close hooks at 1676264027850Disabling compacts and flushes for region at 1676264027850Disabling writes for close at 1676264027855 (+5 ms)Writing region close event to WAL at 1676264027858 (+3 ms)Running coprocessor post-close hooks at 1676264027868 (+10 ms)Closed at 1676264027868 2023-02-13 04:53:47,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676264024198.44f7466753cf1e647dbfa6434db654ad. 2023-02-13 04:53:47,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676264023581.27466f1fdc4879f7b76534c1c3c09a5c. 2023-02-13 04:53:47,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9afdb8e6bb1d6887dd3ec1d87c561bcc, disabling compactions & flushes 2023-02-13 04:53:47,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:47,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:47,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. after waiting 0 ms 2023-02-13 04:53:47,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:47,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9afdb8e6bb1d6887dd3ec1d87c561bcc 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 04:53:47,887 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/.tmp/info/0193b9fdecb04f578673c96a9cccdca8 2023-02-13 04:53:47,903 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/.tmp/info/5e138472b5cc41cea5185d3c118b8d6d 2023-02-13 04:53:47,905 INFO [regionserver/jenkins-hbase4:33925.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:47,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/.tmp/info/5e138472b5cc41cea5185d3c118b8d6d as hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/info/5e138472b5cc41cea5185d3c118b8d6d 2023-02-13 04:53:47,921 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:47,922 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:47,925 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/.tmp/table/8a53cb240cc0431f87c4e1403efcea62 2023-02-13 04:53:47,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/info/5e138472b5cc41cea5185d3c118b8d6d, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 04:53:47,933 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/.tmp/info/0193b9fdecb04f578673c96a9cccdca8 as hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/info/0193b9fdecb04f578673c96a9cccdca8 2023-02-13 04:53:47,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 9afdb8e6bb1d6887dd3ec1d87c561bcc in 64ms, sequenceid=6, compaction requested=false 2023-02-13 04:53:47,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/namespace/9afdb8e6bb1d6887dd3ec1d87c561bcc/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 04:53:47,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:47,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9afdb8e6bb1d6887dd3ec1d87c561bcc: Waiting for close lock at 1676264027870Running coprocessor pre-close hooks at 1676264027870Disabling compacts and flushes for region at 1676264027870Disabling writes for close at 1676264027870Obtaining lock to block concurrent updates at 1676264027871 (+1 ms)Preparing flush snapshotting stores in 9afdb8e6bb1d6887dd3ec1d87c561bcc at 1676264027871Finished memstore snapshotting hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676264027871Flushing stores of hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. at 1676264027872 (+1 ms)Flushing 9afdb8e6bb1d6887dd3ec1d87c561bcc/info: creating writer at 1676264027872Flushing 9afdb8e6bb1d6887dd3ec1d87c561bcc/info: appending metadata at 1676264027876 (+4 ms)Flushing 9afdb8e6bb1d6887dd3ec1d87c561bcc/info: closing flushed file at 1676264027876Flushing 9afdb8e6bb1d6887dd3ec1d87c561bcc/info: reopening flushed file at 1676264027916 (+40 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 9afdb8e6bb1d6887dd3ec1d87c561bcc in 64ms, sequenceid=6, compaction requested=false at 1676264027934 (+18 ms)Writing region close event to WAL at 1676264027936 (+2 ms)Running coprocessor post-close hooks at 1676264027941 (+5 ms)Closed at 1676264027942 (+1 ms) 2023-02-13 04:53:47,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676264022094.9afdb8e6bb1d6887dd3ec1d87c561bcc. 2023-02-13 04:53:47,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f94e6da30a1d28bfa701f135a39ba73a, disabling compactions & flushes 2023-02-13 04:53:47,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:47,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:47,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. after waiting 0 ms 2023-02-13 04:53:47,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:47,943 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/info/0193b9fdecb04f578673c96a9cccdca8, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 04:53:47,945 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/.tmp/table/8a53cb240cc0431f87c4e1403efcea62 as hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/table/8a53cb240cc0431f87c4e1403efcea62 2023-02-13 04:53:47,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/default/restartTableOne/f94e6da30a1d28bfa701f135a39ba73a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 04:53:47,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:47,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f94e6da30a1d28bfa701f135a39ba73a: Waiting for close lock at 1676264027942Running coprocessor pre-close hooks at 1676264027942Disabling compacts and flushes for region at 1676264027942Disabling writes for close at 1676264027943 (+1 ms)Writing region close event to WAL at 1676264027943Running coprocessor post-close hooks at 1676264027948 (+5 ms)Closed at 1676264027949 (+1 ms) 2023-02-13 04:53:47,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676264022940.f94e6da30a1d28bfa701f135a39ba73a. 2023-02-13 04:53:47,953 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/table/8a53cb240cc0431f87c4e1403efcea62, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 04:53:47,954 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 97ms, sequenceid=24, compaction requested=false 2023-02-13 04:53:47,957 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,957 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,957 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,957 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,957 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,957 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33925,1676264027183 2023-02-13 04:53:47,958 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f2f1519 rejected from java.util.concurrent.ThreadPoolExecutor@3b108624[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:47,959 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:47,959 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@8e6cff9 rejected from java.util.concurrent.ThreadPoolExecutor@3b108624[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:47,959 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33925,1676264027183] 2023-02-13 04:53:47,960 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33925,1676264027183; numProcessing=1 2023-02-13 04:53:47,963 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 04:53:47,963 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33925,1676264027183 already deleted, retry=false 2023-02-13 04:53:47,963 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33925,1676264027183 expired; onlineServers=2 2023-02-13 04:53:47,963 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 04:53:47,964 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:47,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264027853Running coprocessor pre-close hooks at 1676264027853Disabling compacts and flushes for region at 1676264027853Disabling writes for close at 1676264027857 (+4 ms)Obtaining lock to block concurrent updates at 1676264027858 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676264027858Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676264027858Flushing stores of hbase:meta,,1.1588230740 at 1676264027860 (+2 ms)Flushing 1588230740/info: creating writer at 1676264027860Flushing 1588230740/info: appending metadata at 1676264027870 (+10 ms)Flushing 1588230740/info: closing flushed file at 1676264027870Flushing 1588230740/table: creating writer at 1676264027903 (+33 ms)Flushing 1588230740/table: appending metadata at 1676264027906 (+3 ms)Flushing 1588230740/table: closing flushed file at 1676264027906Flushing 1588230740/info: reopening flushed file at 1676264027934 (+28 ms)Flushing 1588230740/table: reopening flushed file at 1676264027946 (+12 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 97ms, sequenceid=24, compaction requested=false at 1676264027954 (+8 ms)Writing region close event to WAL at 1676264027958 (+4 ms)Running coprocessor post-close hooks at 1676264027963 (+5 ms)Closed at 1676264027964 (+1 ms) 2023-02-13 04:53:47,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:48,054 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37801,1676264020037; all regions closed. 2023-02-13 04:53:48,054 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36873,1676264020002; all regions closed. 2023-02-13 04:53:48,062 DEBUG [RS:2;jenkins-hbase4:37801] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs 2023-02-13 04:53:48,062 DEBUG [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs 2023-02-13 04:53:48,062 INFO [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36873%2C1676264020002.meta:.meta(num 1676264021792) 2023-02-13 04:53:48,062 INFO [RS:2;jenkins-hbase4:37801] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37801%2C1676264020037:(num 1676264021579) 2023-02-13 04:53:48,062 DEBUG [RS:2;jenkins-hbase4:37801] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:48,062 INFO [RS:2;jenkins-hbase4:37801] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:48,062 INFO [RS:2;jenkins-hbase4:37801] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:48,063 INFO [RS:2;jenkins-hbase4:37801] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:48,063 INFO [RS:2;jenkins-hbase4:37801] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:48,063 INFO [RS:2;jenkins-hbase4:37801] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:48,063 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:48,063 INFO [RS:2;jenkins-hbase4:37801] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37801 2023-02-13 04:53:48,066 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:48,066 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37801,1676264020037 2023-02-13 04:53:48,066 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:48,068 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37801,1676264020037] 2023-02-13 04:53:48,068 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37801,1676264020037; numProcessing=2 2023-02-13 04:53:48,069 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37801,1676264020037 already deleted, retry=false 2023-02-13 04:53:48,069 DEBUG [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/oldWALs 2023-02-13 04:53:48,069 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37801,1676264020037 expired; onlineServers=1 2023-02-13 04:53:48,069 INFO [RS:1;jenkins-hbase4:36873] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36873%2C1676264020002:(num 1676264021579) 2023-02-13 04:53:48,070 DEBUG [RS:1;jenkins-hbase4:36873] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:48,070 INFO [RS:1;jenkins-hbase4:36873] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:48,070 INFO [RS:1;jenkins-hbase4:36873] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:48,070 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:48,070 INFO [RS:1;jenkins-hbase4:36873] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36873 2023-02-13 04:53:48,077 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36873,1676264020002 2023-02-13 04:53:48,077 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:48,077 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43ee90bc rejected from java.util.concurrent.ThreadPoolExecutor@47d10c19[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,079 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36873,1676264020002] 2023-02-13 04:53:48,079 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36873,1676264020002; numProcessing=3 2023-02-13 04:53:48,080 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36873,1676264020002 already deleted, retry=false 2023-02-13 04:53:48,080 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36873,1676264020002 expired; onlineServers=0 2023-02-13 04:53:48,080 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36759,1676264027143' ***** 2023-02-13 04:53:48,080 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 04:53:48,080 DEBUG [M:0;jenkins-hbase4:36759] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@777be5f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:48,080 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36759,1676264027143 2023-02-13 04:53:48,080 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36759,1676264027143; all regions closed. 2023-02-13 04:53:48,080 DEBUG [M:0;jenkins-hbase4:36759] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:48,080 DEBUG [M:0;jenkins-hbase4:36759] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 04:53:48,080 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 04:53:48,080 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264027444] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264027444,5,FailOnTimeoutGroup] 2023-02-13 04:53:48,080 DEBUG [M:0;jenkins-hbase4:36759] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 04:53:48,081 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264027444] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264027444,5,FailOnTimeoutGroup] 2023-02-13 04:53:48,081 INFO [M:0;jenkins-hbase4:36759] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 04:53:48,082 INFO [M:0;jenkins-hbase4:36759] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 04:53:48,082 INFO [M:0;jenkins-hbase4:36759] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 04:53:48,082 DEBUG [M:0;jenkins-hbase4:36759] master.HMaster(1502): Stopping service threads 2023-02-13 04:53:48,082 INFO [M:0;jenkins-hbase4:36759] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 04:53:48,082 ERROR [M:0;jenkins-hbase4:36759] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 04:53:48,082 INFO [M:0;jenkins-hbase4:36759] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 04:53:48,082 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:48,082 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 04:53:48,082 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:48,083 DEBUG [M:0;jenkins-hbase4:36759] zookeeper.ZKUtil(398): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 04:53:48,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:48,083 WARN [M:0;jenkins-hbase4:36759] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 04:53:48,083 INFO [M:0;jenkins-hbase4:36759] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 04:53:48,083 INFO [M:0;jenkins-hbase4:36759] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 04:53:48,084 DEBUG [M:0;jenkins-hbase4:36759] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 04:53:48,084 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:48,084 DEBUG [M:0;jenkins-hbase4:36759] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:48,084 DEBUG [M:0;jenkins-hbase4:36759] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 04:53:48,084 DEBUG [M:0;jenkins-hbase4:36759] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:48,084 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=2.98 KB heapSize=3.97 KB 2023-02-13 04:53:48,105 INFO [M:0;jenkins-hbase4:36759] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.98 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d8d071402e99460ebda523ebb506802d 2023-02-13 04:53:48,113 DEBUG [M:0;jenkins-hbase4:36759] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d8d071402e99460ebda523ebb506802d as hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d8d071402e99460ebda523ebb506802d 2023-02-13 04:53:48,121 INFO [M:0;jenkins-hbase4:36759] regionserver.HStore(1080): Added hdfs://localhost:37257/user/jenkins/test-data/a956736d-069e-959e-300e-8b1fc696e56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d8d071402e99460ebda523ebb506802d, entries=1, sequenceid=155, filesize=5.1 K 2023-02-13 04:53:48,124 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegion(2947): Finished flush of dataSize ~2.98 KB/3050, heapSize ~3.95 KB/4048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 40ms, sequenceid=155, compaction requested=false 2023-02-13 04:53:48,127 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:48,127 DEBUG [M:0;jenkins-hbase4:36759] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264028083Disabling compacts and flushes for region at 1676264028083Disabling writes for close at 1676264028084 (+1 ms)Obtaining lock to block concurrent updates at 1676264028084Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676264028084Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=3050, getHeapSize=4048, getOffHeapSize=0, getCellsCount=10 at 1676264028084Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676264028085 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676264028086 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676264028088 (+2 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676264028089 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676264028114 (+25 ms)Finished flush of dataSize ~2.98 KB/3050, heapSize ~3.95 KB/4048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 40ms, sequenceid=155, compaction requested=false at 1676264028124 (+10 ms)Writing region close event to WAL at 1676264028127 (+3 ms)Closed at 1676264028127 2023-02-13 04:53:48,132 INFO [M:0;jenkins-hbase4:36759] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 04:53:48,132 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:48,133 INFO [M:0;jenkins-hbase4:36759] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36759 2023-02-13 04:53:48,136 DEBUG [M:0;jenkins-hbase4:36759] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,36759,1676264027143 already deleted, retry=false 2023-02-13 04:53:48,344 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,344 INFO [M:0;jenkins-hbase4:36759] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36759,1676264027143; zookeeper connection closed. 2023-02-13 04:53:48,344 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6d49a768 rejected from java.util.concurrent.ThreadPoolExecutor@69762e19[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,345 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): master:36759-0x1004c06f9cb0009, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,345 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3d4000b9 rejected from java.util.concurrent.ThreadPoolExecutor@69762e19[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,444 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,444 INFO [RS:1;jenkins-hbase4:36873] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36873,1676264020002; zookeeper connection closed. 2023-02-13 04:53:48,444 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c7b79b rejected from java.util.concurrent.ThreadPoolExecutor@47d10c19[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,445 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3b41688a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3b41688a 2023-02-13 04:53:48,445 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:36873-0x1004c06f9cb0002, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,445 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ce5cf71 rejected from java.util.concurrent.ThreadPoolExecutor@47d10c19[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,544 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,544 INFO [RS:2;jenkins-hbase4:37801] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37801,1676264020037; zookeeper connection closed. 2023-02-13 04:53:48,545 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@314d72ca rejected from java.util.concurrent.ThreadPoolExecutor@3572f03b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,545 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7ec156e3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7ec156e3 2023-02-13 04:53:48,545 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:37801-0x1004c06f9cb0003, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,545 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1bcbe577 rejected from java.util.concurrent.ThreadPoolExecutor@3572f03b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,645 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,645 INFO [RS:3;jenkins-hbase4:33925] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33925,1676264027183; zookeeper connection closed. 2023-02-13 04:53:48,645 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@70b80a57 rejected from java.util.concurrent.ThreadPoolExecutor@3b108624[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,645 DEBUG [Listener at localhost/41311-EventThread] zookeeper.ZKWatcher(600): regionserver:33925-0x1004c06f9cb000a, quorum=127.0.0.1:52493, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:48,645 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@57753502] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@57753502 2023-02-13 04:53:48,645 ERROR [Listener at localhost/41311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@201a7d54 rejected from java.util.concurrent.ThreadPoolExecutor@3b108624[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:48,646 INFO [Listener at localhost/41311] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 4 regionserver(s) complete 2023-02-13 04:53:48,646 WARN [Listener at localhost/41311] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 04:53:48,651 INFO [Listener at localhost/41311] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:53:48,755 WARN [BP-409205819-172.31.14.131-1676264015148 heartbeating to localhost/127.0.0.1:37257] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 04:53:48,755 WARN [BP-409205819-172.31.14.131-1676264015148 heartbeating to localhost/127.0.0.1:37257] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-409205819-172.31.14.131-1676264015148 (Datanode Uuid 4625348a-b30e-4cb4-ba12-9854c58ab386) service to localhost/127.0.0.1:37257 2023-02-13 04:53:48,757 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/dfs/data/data5/current/BP-409205819-172.31.14.131-1676264015148] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:48,758 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/dfs/data/data6/current/BP-409205819-172.31.14.131-1676264015148] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:48,759 WARN [Listener at localhost/41311] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 04:53:48,761 INFO [Listener at localhost/41311] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:53:48,863 WARN [BP-409205819-172.31.14.131-1676264015148 heartbeating to localhost/127.0.0.1:37257] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 04:53:48,863 WARN [BP-409205819-172.31.14.131-1676264015148 heartbeating to localhost/127.0.0.1:37257] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-409205819-172.31.14.131-1676264015148 (Datanode Uuid 6ed8807f-8601-499f-9051-a604c2a88c5d) service to localhost/127.0.0.1:37257 2023-02-13 04:53:48,864 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/dfs/data/data3/current/BP-409205819-172.31.14.131-1676264015148] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:48,864 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/dfs/data/data4/current/BP-409205819-172.31.14.131-1676264015148] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:48,866 WARN [Listener at localhost/41311] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 04:53:48,868 INFO [Listener at localhost/41311] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:53:48,970 WARN [BP-409205819-172.31.14.131-1676264015148 heartbeating to localhost/127.0.0.1:37257] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 04:53:48,970 WARN [BP-409205819-172.31.14.131-1676264015148 heartbeating to localhost/127.0.0.1:37257] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-409205819-172.31.14.131-1676264015148 (Datanode Uuid 0c1e1f28-bae9-4953-b26e-014656334ec1) service to localhost/127.0.0.1:37257 2023-02-13 04:53:48,971 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/dfs/data/data1/current/BP-409205819-172.31.14.131-1676264015148] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:48,971 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/cluster_14d48cb9-c37c-8ca4-1c53-ad9de9a9dc43/dfs/data/data2/current/BP-409205819-172.31.14.131-1676264015148] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:49,002 INFO [Listener at localhost/41311] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:53:49,119 INFO [Listener at localhost/41311] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 04:53:49,177 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 04:53:49,192 INFO [Listener at localhost/41311] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=101 (was 7) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:37257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/41311 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: regionserver/jenkins-hbase4:33925.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:37257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:37257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: ForkJoinPool-2-worker-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:37257 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@678b7573 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:37257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:37257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:37257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=504 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=320 (was 289) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4218 (was 5365) 2023-02-13 04:53:49,203 INFO [Listener at localhost/41311] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=101, OpenFileDescriptor=504, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=168, AvailableMemoryMB=4218 2023-02-13 04:53:49,214 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/hadoop.log.dir so I do NOT create it in target/test-data/792f905b-e817-3924-a7de-d18feef7e991 2023-02-13 04:53:49,214 WARN [Listener at localhost/41311] hbase.HBaseTestingUtility(449): hadoop.log.dir property value differs in configuration and system: Configuration=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/../logs while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/hadoop.log.dir Erasing configuration value by system value. 2023-02-13 04:53:49,214 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/hadoop.tmp.dir so I do NOT create it in target/test-data/792f905b-e817-3924-a7de-d18feef7e991 2023-02-13 04:53:49,214 WARN [Listener at localhost/41311] hbase.HBaseTestingUtility(449): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-jenkins while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ae4bc50b-4795-f731-dd70-27373723e396/hadoop.tmp.dir Erasing configuration value by system value. 2023-02-13 04:53:49,214 DEBUG [Listener at localhost/41311] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991 2023-02-13 04:53:49,214 INFO [Listener at localhost/41311] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7, deleteOnExit=true 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/test.cache.data in system properties and HBase conf 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/hadoop.log.dir in system properties and HBase conf 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 04:53:49,215 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 04:53:49,216 DEBUG [Listener at localhost/41311] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 04:53:49,216 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 04:53:49,216 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 04:53:49,216 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 04:53:49,216 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 04:53:49,216 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 04:53:49,216 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/nfs.dump.dir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/java.io.tmpdir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 04:53:49,217 INFO [Listener at localhost/41311] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 04:53:49,223 WARN [Listener at localhost/41311] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 04:53:49,223 WARN [Listener at localhost/41311] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 04:53:49,268 WARN [Listener at localhost/41311] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:49,271 INFO [Listener at localhost/41311] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:49,276 INFO [Listener at localhost/41311] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/java.io.tmpdir/Jetty_localhost_35975_hdfs____.oft735/webapp 2023-02-13 04:53:49,372 INFO [Listener at localhost/41311] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35975 2023-02-13 04:53:49,382 WARN [Listener at localhost/41311] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 04:53:49,382 WARN [Listener at localhost/41311] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 04:53:49,435 WARN [Listener at localhost/44887] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:49,452 WARN [Listener at localhost/44887] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 04:53:49,455 WARN [Listener at localhost/44887] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:49,456 INFO [Listener at localhost/44887] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:49,462 INFO [Listener at localhost/44887] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/java.io.tmpdir/Jetty_localhost_42543_datanode____1vqrdp/webapp 2023-02-13 04:53:49,555 INFO [Listener at localhost/44887] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42543 2023-02-13 04:53:49,563 WARN [Listener at localhost/34955] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:49,580 WARN [Listener at localhost/34955] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 04:53:49,582 WARN [Listener at localhost/34955] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:49,583 INFO [Listener at localhost/34955] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:49,587 INFO [Listener at localhost/34955] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/java.io.tmpdir/Jetty_localhost_46849_datanode____dsh0ma/webapp 2023-02-13 04:53:49,680 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6da8ad70b764e3aa: Processing first storage report for DS-2db251de-67f9-46dc-9c0a-39f27904fc17 from datanode 1010f7f1-03e6-4a21-9a30-9dde67a8be02 2023-02-13 04:53:49,680 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6da8ad70b764e3aa: from storage DS-2db251de-67f9-46dc-9c0a-39f27904fc17 node DatanodeRegistration(127.0.0.1:43657, datanodeUuid=1010f7f1-03e6-4a21-9a30-9dde67a8be02, infoPort=45079, infoSecurePort=0, ipcPort=34955, storageInfo=lv=-57;cid=testClusterID;nsid=422454637;c=1676264029226), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 04:53:49,680 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6da8ad70b764e3aa: Processing first storage report for DS-022c5785-474b-4d5e-b800-4f94fb623f23 from datanode 1010f7f1-03e6-4a21-9a30-9dde67a8be02 2023-02-13 04:53:49,680 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6da8ad70b764e3aa: from storage DS-022c5785-474b-4d5e-b800-4f94fb623f23 node DatanodeRegistration(127.0.0.1:43657, datanodeUuid=1010f7f1-03e6-4a21-9a30-9dde67a8be02, infoPort=45079, infoSecurePort=0, ipcPort=34955, storageInfo=lv=-57;cid=testClusterID;nsid=422454637;c=1676264029226), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:49,687 INFO [Listener at localhost/34955] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46849 2023-02-13 04:53:49,697 WARN [Listener at localhost/35113] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:49,714 WARN [Listener at localhost/35113] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 04:53:49,716 WARN [Listener at localhost/35113] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 04:53:49,718 INFO [Listener at localhost/35113] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 04:53:49,763 INFO [Listener at localhost/35113] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/java.io.tmpdir/Jetty_localhost_42897_datanode____fgp05b/webapp 2023-02-13 04:53:49,835 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x517b788357e0efcb: Processing first storage report for DS-6f049793-407a-4f8d-aa70-9720627cd3e0 from datanode 3da01103-3cef-49b3-8ff9-04fda312a49c 2023-02-13 04:53:49,835 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x517b788357e0efcb: from storage DS-6f049793-407a-4f8d-aa70-9720627cd3e0 node DatanodeRegistration(127.0.0.1:39881, datanodeUuid=3da01103-3cef-49b3-8ff9-04fda312a49c, infoPort=46083, infoSecurePort=0, ipcPort=35113, storageInfo=lv=-57;cid=testClusterID;nsid=422454637;c=1676264029226), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:49,835 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x517b788357e0efcb: Processing first storage report for DS-a47b670d-0a48-4d3e-a338-3c22ab1c6d42 from datanode 3da01103-3cef-49b3-8ff9-04fda312a49c 2023-02-13 04:53:49,835 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x517b788357e0efcb: from storage DS-a47b670d-0a48-4d3e-a338-3c22ab1c6d42 node DatanodeRegistration(127.0.0.1:39881, datanodeUuid=3da01103-3cef-49b3-8ff9-04fda312a49c, infoPort=46083, infoSecurePort=0, ipcPort=35113, storageInfo=lv=-57;cid=testClusterID;nsid=422454637;c=1676264029226), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:49,871 INFO [Listener at localhost/35113] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42897 2023-02-13 04:53:49,882 WARN [Listener at localhost/33627] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 04:53:49,986 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbba320bb1bca68c4: Processing first storage report for DS-adb1b923-3f68-4792-94e0-90ea0aea59c2 from datanode 709dad92-a625-4c99-8c68-4c7530104616 2023-02-13 04:53:49,986 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbba320bb1bca68c4: from storage DS-adb1b923-3f68-4792-94e0-90ea0aea59c2 node DatanodeRegistration(127.0.0.1:39251, datanodeUuid=709dad92-a625-4c99-8c68-4c7530104616, infoPort=32941, infoSecurePort=0, ipcPort=33627, storageInfo=lv=-57;cid=testClusterID;nsid=422454637;c=1676264029226), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:49,986 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbba320bb1bca68c4: Processing first storage report for DS-58049c45-7817-4372-bd2c-f7902c6cffef from datanode 709dad92-a625-4c99-8c68-4c7530104616 2023-02-13 04:53:49,986 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbba320bb1bca68c4: from storage DS-58049c45-7817-4372-bd2c-f7902c6cffef node DatanodeRegistration(127.0.0.1:39251, datanodeUuid=709dad92-a625-4c99-8c68-4c7530104616, infoPort=32941, infoSecurePort=0, ipcPort=33627, storageInfo=lv=-57;cid=testClusterID;nsid=422454637;c=1676264029226), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 04:53:50,015 DEBUG [Listener at localhost/33627] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991 2023-02-13 04:53:50,020 INFO [Listener at localhost/33627] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/zookeeper_0, clientPort=59910, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 04:53:50,023 INFO [Listener at localhost/33627] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59910 2023-02-13 04:53:50,023 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,024 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,048 INFO [Listener at localhost/33627] util.FSUtils(479): Created version file at hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec with version=8 2023-02-13 04:53:50,049 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/hbase-staging 2023-02-13 04:53:50,050 INFO [Listener at localhost/33627] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:50,051 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,051 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,051 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:50,051 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,051 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:50,051 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:50,053 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36177 2023-02-13 04:53:50,053 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,054 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,055 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36177 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:50,069 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:361770x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:50,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36177-0x1004c0729060000 connected 2023-02-13 04:53:50,087 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:50,087 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:50,088 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:50,092 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36177 2023-02-13 04:53:50,092 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36177 2023-02-13 04:53:50,092 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36177 2023-02-13 04:53:50,092 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36177 2023-02-13 04:53:50,093 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36177 2023-02-13 04:53:50,093 INFO [Listener at localhost/33627] master.HMaster(439): hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec, hbase.cluster.distributed=false 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:50,106 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:50,107 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40715 2023-02-13 04:53:50,107 INFO [Listener at localhost/33627] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:50,109 DEBUG [Listener at localhost/33627] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:50,109 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,110 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,111 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40715 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:50,115 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:407150x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:50,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40715-0x1004c0729060001 connected 2023-02-13 04:53:50,116 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:50,117 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:50,117 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:50,118 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40715 2023-02-13 04:53:50,118 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40715 2023-02-13 04:53:50,118 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40715 2023-02-13 04:53:50,119 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40715 2023-02-13 04:53:50,119 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40715 2023-02-13 04:53:50,129 INFO [Listener at localhost/33627] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:50,130 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,130 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,130 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:50,130 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,130 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:50,130 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:50,131 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40473 2023-02-13 04:53:50,131 INFO [Listener at localhost/33627] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:50,132 DEBUG [Listener at localhost/33627] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:50,133 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,134 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,135 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40473 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:50,138 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:404730x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:50,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40473-0x1004c0729060002 connected 2023-02-13 04:53:50,140 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:50,140 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:50,141 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:50,141 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40473 2023-02-13 04:53:50,142 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40473 2023-02-13 04:53:50,142 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40473 2023-02-13 04:53:50,142 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40473 2023-02-13 04:53:50,143 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40473 2023-02-13 04:53:50,153 INFO [Listener at localhost/33627] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:50,153 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,154 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,154 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:50,154 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:50,154 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:50,154 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:50,155 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34837 2023-02-13 04:53:50,155 INFO [Listener at localhost/33627] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:50,156 DEBUG [Listener at localhost/33627] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:50,156 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,157 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,158 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34837 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:50,161 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:348370x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:50,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34837-0x1004c0729060003 connected 2023-02-13 04:53:50,163 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:50,164 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:50,164 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:50,167 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34837 2023-02-13 04:53:50,167 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34837 2023-02-13 04:53:50,168 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34837 2023-02-13 04:53:50,168 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34837 2023-02-13 04:53:50,168 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34837 2023-02-13 04:53:50,170 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,172 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:50,172 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,175 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:50,175 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:50,175 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:50,175 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:50,175 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:50,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:50,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,36177,1676264030050 from backup master directory 2023-02-13 04:53:50,180 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,180 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:50,180 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:50,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,203 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/hbase.id with ID: 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:50,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:50,220 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6bf83df4 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:50,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e1a1a35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:50,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:50,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 04:53:50,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:50,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store-tmp 2023-02-13 04:53:50,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:50,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 04:53:50,253 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:50,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:50,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 04:53:50,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:50,253 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:50,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264030253Disabling compacts and flushes for region at 1676264030253Disabling writes for close at 1676264030253Writing region close event to WAL at 1676264030253Closed at 1676264030253 2023-02-13 04:53:50,254 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,257 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36177%2C1676264030050, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/oldWALs, maxLogs=10 2023-02-13 04:53:50,272 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:50,274 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:50,272 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:50,276 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 2023-02-13 04:53:50,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK], DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK]] 2023-02-13 04:53:50,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:50,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:50,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:50,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:50,282 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:50,284 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 04:53:50,285 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 04:53:50,285 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:50,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:50,290 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:50,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:50,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74576793, jitterRate=0.11128081381320953}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:50,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676264030280Initializing all the Stores at 1676264030281 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264030281Cleaning up temporary data from old regions at 1676264030287 (+6 ms)Cleaning up detritus from prior splits at 1676264030288 (+1 ms)Region opened successfully at 1676264030292 (+4 ms) 2023-02-13 04:53:50,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 04:53:50,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 04:53:50,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 04:53:50,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 04:53:50,298 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 04:53:50,298 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-13 04:53:50,298 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 04:53:50,299 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 04:53:50,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 04:53:50,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 04:53:50,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 04:53:50,312 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 04:53:50,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 04:53:50,312 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 04:53:50,315 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 04:53:50,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 04:53:50,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 04:53:50,319 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:50,319 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:50,319 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:50,319 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,319 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:50,320 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,36177,1676264030050, sessionid=0x1004c0729060000, setting cluster-up flag (Was=false) 2023-02-13 04:53:50,323 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 04:53:50,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,335 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 04:53:50,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:50,341 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.hbase-snapshot/.tmp 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:50,347 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676264060352 2023-02-13 04:53:50,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 04:53:50,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 04:53:50,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 04:53:50,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 04:53:50,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 04:53:50,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 04:53:50,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,353 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:50,353 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 04:53:50,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 04:53:50,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 04:53:50,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 04:53:50,355 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:50,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 04:53:50,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 04:53:50,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264030356,5,FailOnTimeoutGroup] 2023-02-13 04:53:50,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264030358,5,FailOnTimeoutGroup] 2023-02-13 04:53:50,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 04:53:50,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,371 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:50,371 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:50,372 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:50,373 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:50,373 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:50,373 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:50,378 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:50,378 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:50,378 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:50,378 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:50,379 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:50,379 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:50,379 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:50,379 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:50,379 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:50,381 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:50,382 DEBUG [RS:1;jenkins-hbase4:40473] zookeeper.ReadOnlyZKClient(139): Connect 0x521f8f41 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:50,382 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:50,383 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:50,383 DEBUG [RS:0;jenkins-hbase4:40715] zookeeper.ReadOnlyZKClient(139): Connect 0x0a9c6b68 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:50,385 DEBUG [RS:2;jenkins-hbase4:34837] zookeeper.ReadOnlyZKClient(139): Connect 0x6bc6ce21 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:50,401 DEBUG [RS:1;jenkins-hbase4:40473] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cd87b72, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:50,402 DEBUG [RS:1;jenkins-hbase4:40473] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1de57149, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:50,408 DEBUG [RS:2;jenkins-hbase4:34837] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65013d29, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:50,408 DEBUG [RS:0;jenkins-hbase4:40715] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45b944b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:50,408 DEBUG [RS:2;jenkins-hbase4:34837] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@613ea551, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:50,408 DEBUG [RS:0;jenkins-hbase4:40715] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@640018, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:50,417 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:40473 2023-02-13 04:53:50,418 INFO [RS:1;jenkins-hbase4:40473] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:50,418 INFO [RS:1;jenkins-hbase4:40473] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:50,418 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:50,419 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36177,1676264030050 with isa=jenkins-hbase4.apache.org/172.31.14.131:40473, startcode=1676264030129 2023-02-13 04:53:50,419 DEBUG [RS:1;jenkins-hbase4:40473] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:50,424 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41883, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:50,424 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:50,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,427 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:50,427 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44887 2023-02-13 04:53:50,427 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:34837 2023-02-13 04:53:50,427 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:50,427 INFO [RS:2;jenkins-hbase4:34837] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:50,427 INFO [RS:2;jenkins-hbase4:34837] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:50,427 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:50,428 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:40715 2023-02-13 04:53:50,428 INFO [RS:0;jenkins-hbase4:40715] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:50,428 INFO [RS:0;jenkins-hbase4:40715] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:50,428 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:50,436 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 04:53:50,436 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:50,437 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36177,1676264030050 with isa=jenkins-hbase4.apache.org/172.31.14.131:40715, startcode=1676264030105 2023-02-13 04:53:50,437 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,36177,1676264030050 with isa=jenkins-hbase4.apache.org/172.31.14.131:34837, startcode=1676264030153 2023-02-13 04:53:50,437 DEBUG [RS:0;jenkins-hbase4:40715] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:50,438 DEBUG [RS:2;jenkins-hbase4:34837] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:50,438 DEBUG [RS:1;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,438 WARN [RS:1;jenkins-hbase4:40473] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:50,438 INFO [RS:1;jenkins-hbase4:40473] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:50,438 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40473,1676264030129] 2023-02-13 04:53:50,438 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,439 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info 2023-02-13 04:53:50,439 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 04:53:50,440 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37599, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:50,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,441 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 04:53:50,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,440 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43557, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.7 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:50,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,442 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:50,442 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44887 2023-02-13 04:53:50,442 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:50,442 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:50,442 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44887 2023-02-13 04:53:50,443 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:50,444 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:50,445 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 04:53:50,445 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,446 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 04:53:50,447 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table 2023-02-13 04:53:50,448 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:50,448 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40715,1676264030105] 2023-02-13 04:53:50,449 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34837,1676264030153] 2023-02-13 04:53:50,449 DEBUG [RS:0;jenkins-hbase4:40715] zookeeper.ZKUtil(162): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,449 WARN [RS:0;jenkins-hbase4:40715] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:50,449 INFO [RS:0;jenkins-hbase4:40715] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:50,449 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,449 DEBUG [RS:2;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,449 WARN [RS:2;jenkins-hbase4:34837] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:50,449 INFO [RS:2;jenkins-hbase4:34837] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:50,449 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 04:53:50,449 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,450 DEBUG [RS:1;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,450 DEBUG [RS:1;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,450 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,450 DEBUG [RS:1;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,451 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:50,451 INFO [RS:1;jenkins-hbase4:40473] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:50,457 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740 2023-02-13 04:53:50,459 INFO [RS:1;jenkins-hbase4:40473] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:50,460 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740 2023-02-13 04:53:50,461 INFO [RS:1;jenkins-hbase4:40473] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:50,461 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,464 DEBUG [RS:0;jenkins-hbase4:40715] zookeeper.ZKUtil(162): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,464 DEBUG [RS:0;jenkins-hbase4:40715] zookeeper.ZKUtil(162): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,465 DEBUG [RS:0;jenkins-hbase4:40715] zookeeper.ZKUtil(162): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,466 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 04:53:50,468 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:50,468 INFO [RS:0;jenkins-hbase4:40715] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:50,470 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:50,471 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 04:53:50,472 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,472 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,472 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,472 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,473 DEBUG [RS:1;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,475 DEBUG [RS:2;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,476 DEBUG [RS:2;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,476 DEBUG [RS:2;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,477 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:50,477 INFO [RS:2;jenkins-hbase4:34837] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:50,480 INFO [RS:0;jenkins-hbase4:40715] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:50,487 INFO [RS:2;jenkins-hbase4:34837] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:50,492 INFO [RS:0;jenkins-hbase4:40715] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:50,492 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,492 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,497 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:50,497 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,497 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:50,497 INFO [RS:2;jenkins-hbase4:34837] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:50,498 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,497 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,499 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67641583, jitterRate=0.00793813169002533}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 04:53:50,499 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676264030424Initializing all the Stores at 1676264030427 (+3 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264030427Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264030435 (+8 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264030435Cleaning up temporary data from old regions at 1676264030464 (+29 ms)Cleaning up detritus from prior splits at 1676264030465 (+1 ms)Region opened successfully at 1676264030499 (+34 ms) 2023-02-13 04:53:50,499 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 04:53:50,499 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 04:53:50,499 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 04:53:50,499 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 04:53:50,499 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 04:53:50,499 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:50,505 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:50,506 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,506 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,507 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:50,506 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,507 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264030499Disabling compacts and flushes for region at 1676264030499Disabling writes for close at 1676264030499Writing region close event to WAL at 1676264030507 (+8 ms)Closed at 1676264030507 2023-02-13 04:53:50,507 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:50,507 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,507 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,507 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,507 DEBUG [RS:2;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,507 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,508 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,508 DEBUG [RS:0;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:50,509 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:50,509 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 04:53:50,509 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 04:53:50,510 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,510 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,510 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,516 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 04:53:50,524 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,524 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,524 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,525 INFO [RS:1;jenkins-hbase4:40473] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:50,525 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40473,1676264030129-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,526 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 04:53:50,533 INFO [RS:2;jenkins-hbase4:34837] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:50,533 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34837,1676264030153-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,539 INFO [RS:1;jenkins-hbase4:40473] regionserver.Replication(203): jenkins-hbase4.apache.org,40473,1676264030129 started 2023-02-13 04:53:50,539 INFO [RS:0;jenkins-hbase4:40715] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:50,539 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40473,1676264030129, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40473, sessionid=0x1004c0729060002 2023-02-13 04:53:50,540 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40715,1676264030105-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,540 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:50,540 DEBUG [RS:1;jenkins-hbase4:40473] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,540 DEBUG [RS:1;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40473,1676264030129' 2023-02-13 04:53:50,540 DEBUG [RS:1;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:50,541 DEBUG [RS:1;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:50,541 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:50,541 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:50,541 DEBUG [RS:1;jenkins-hbase4:40473] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:50,541 DEBUG [RS:1;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40473,1676264030129' 2023-02-13 04:53:50,541 DEBUG [RS:1;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:50,542 DEBUG [RS:1;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:50,542 DEBUG [RS:1;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:50,542 INFO [RS:1;jenkins-hbase4:40473] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:50,542 INFO [RS:1;jenkins-hbase4:40473] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:50,546 INFO [RS:2;jenkins-hbase4:34837] regionserver.Replication(203): jenkins-hbase4.apache.org,34837,1676264030153 started 2023-02-13 04:53:50,546 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34837,1676264030153, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34837, sessionid=0x1004c0729060003 2023-02-13 04:53:50,546 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:50,546 DEBUG [RS:2;jenkins-hbase4:34837] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,546 DEBUG [RS:2;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34837,1676264030153' 2023-02-13 04:53:50,546 DEBUG [RS:2;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34837,1676264030153' 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:50,547 DEBUG [RS:2;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:50,548 DEBUG [RS:2;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:50,548 INFO [RS:2;jenkins-hbase4:34837] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:50,548 INFO [RS:2;jenkins-hbase4:34837] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:50,556 INFO [RS:0;jenkins-hbase4:40715] regionserver.Replication(203): jenkins-hbase4.apache.org,40715,1676264030105 started 2023-02-13 04:53:50,556 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40715,1676264030105, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40715, sessionid=0x1004c0729060001 2023-02-13 04:53:50,556 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:50,556 DEBUG [RS:0;jenkins-hbase4:40715] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,556 DEBUG [RS:0;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40715,1676264030105' 2023-02-13 04:53:50,556 DEBUG [RS:0;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:50,557 DEBUG [RS:0;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:50,557 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:50,557 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:50,557 DEBUG [RS:0;jenkins-hbase4:40715] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:50,557 DEBUG [RS:0;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40715,1676264030105' 2023-02-13 04:53:50,557 DEBUG [RS:0;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:50,558 DEBUG [RS:0;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:50,558 DEBUG [RS:0;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:50,558 INFO [RS:0;jenkins-hbase4:40715] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:50,558 INFO [RS:0;jenkins-hbase4:40715] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:50,645 INFO [RS:1;jenkins-hbase4:40473] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40473%2C1676264030129, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:50,649 INFO [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34837%2C1676264030153, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:50,662 INFO [RS:0;jenkins-hbase4:40715] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40715%2C1676264030105, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:50,664 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:50,668 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:50,670 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:50,678 DEBUG [jenkins-hbase4:36177] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 04:53:50,678 DEBUG [jenkins-hbase4:36177] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:50,679 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:50,679 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:50,679 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:50,680 INFO [RS:1;jenkins-hbase4:40473] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129/jenkins-hbase4.apache.org%2C40473%2C1676264030129.1676264030646 2023-02-13 04:53:50,681 DEBUG [jenkins-hbase4:36177] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:50,681 DEBUG [jenkins-hbase4:36177] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:50,681 DEBUG [jenkins-hbase4:36177] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:50,681 DEBUG [jenkins-hbase4:36177] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:50,681 INFO [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153/jenkins-hbase4.apache.org%2C34837%2C1676264030153.1676264030650 2023-02-13 04:53:50,684 DEBUG [RS:1;jenkins-hbase4:40473] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK], DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK]] 2023-02-13 04:53:50,685 DEBUG [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK]] 2023-02-13 04:53:50,685 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34837,1676264030153, state=OPENING 2023-02-13 04:53:50,687 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 04:53:50,688 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:50,688 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:50,689 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:50,689 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:50,689 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34837,1676264030153}] 2023-02-13 04:53:50,690 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 04:53:50,695 INFO [RS:0;jenkins-hbase4:40715] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105/jenkins-hbase4.apache.org%2C40715%2C1676264030105.1676264030664 2023-02-13 04:53:50,696 DEBUG [RS:0;jenkins-hbase4:40715] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK]] 2023-02-13 04:53:50,846 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:50,846 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:50,848 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40426, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 04:53:50,852 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 04:53:50,852 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:50,854 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34837%2C1676264030153.meta, suffix=.meta, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:50,868 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:50,868 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:50,868 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:50,870 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153/jenkins-hbase4.apache.org%2C34837%2C1676264030153.meta.1676264030854.meta 2023-02-13 04:53:50,871 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK]] 2023-02-13 04:53:50,871 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:50,872 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 04:53:50,872 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 04:53:50,873 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 04:53:50,873 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 04:53:50,873 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:50,873 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 04:53:50,873 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 04:53:50,874 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 04:53:50,875 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info 2023-02-13 04:53:50,875 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info 2023-02-13 04:53:50,876 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 04:53:50,877 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,877 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 04:53:50,878 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:50,878 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:50,879 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 04:53:50,879 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,879 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 04:53:50,880 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table 2023-02-13 04:53:50,880 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table 2023-02-13 04:53:50,880 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 04:53:50,881 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:50,882 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740 2023-02-13 04:53:50,883 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740 2023-02-13 04:53:50,885 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 04:53:50,886 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 04:53:50,887 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73299417, jitterRate=0.09224642813205719}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 04:53:50,887 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676264030873Writing region info on filesystem at 1676264030873Initializing all the Stores at 1676264030874 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264030874Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264030874Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264030874Cleaning up temporary data from old regions at 1676264030884 (+10 ms)Cleaning up detritus from prior splits at 1676264030884Running coprocessor post-open hooks at 1676264030887 (+3 ms)Region opened successfully at 1676264030887 2023-02-13 04:53:50,888 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676264030846 2023-02-13 04:53:50,893 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 04:53:50,894 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 04:53:50,894 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34837,1676264030153, state=OPEN 2023-02-13 04:53:50,898 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 04:53:50,899 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 04:53:50,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 04:53:50,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34837,1676264030153 in 209 msec 2023-02-13 04:53:50,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 04:53:50,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 393 msec 2023-02-13 04:53:50,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 562 msec 2023-02-13 04:53:50,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676264030173, completionTime=-1 2023-02-13 04:53:50,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 04:53:50,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 04:53:50,914 DEBUG [hconnection-0x3fdbf682-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:50,916 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40442, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:50,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 04:53:50,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676264090918 2023-02-13 04:53:50,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676264150918 2023-02-13 04:53:50,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 9 msec 2023-02-13 04:53:50,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36177,1676264030050-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36177,1676264030050-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36177,1676264030050-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:36177, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:50,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 04:53:50,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:50,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 04:53:50,932 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 04:53:50,933 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:50,935 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:50,938 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:50,938 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 empty. 2023-02-13 04:53:50,939 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:50,939 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 04:53:50,958 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:50,959 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => b37bc85591cde75b9b23b10dea6af810, NAME => 'hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp 2023-02-13 04:53:50,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:50,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing b37bc85591cde75b9b23b10dea6af810, disabling compactions & flushes 2023-02-13 04:53:50,972 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:50,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:50,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. after waiting 0 ms 2023-02-13 04:53:50,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:50,972 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:50,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for b37bc85591cde75b9b23b10dea6af810: Waiting for close lock at 1676264030972Disabling compacts and flushes for region at 1676264030972Disabling writes for close at 1676264030972Writing region close event to WAL at 1676264030972Closed at 1676264030972 2023-02-13 04:53:50,975 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:50,976 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264030976"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264030976"}]},"ts":"1676264030976"} 2023-02-13 04:53:50,979 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:50,981 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:50,981 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264030981"}]},"ts":"1676264030981"} 2023-02-13 04:53:50,984 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 04:53:50,987 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:50,988 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:50,988 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:50,988 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:50,988 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:50,988 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN}] 2023-02-13 04:53:50,992 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN 2023-02-13 04:53:50,993 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40473,1676264030129; forceNewPlan=false, retain=false 2023-02-13 04:53:51,143 INFO [jenkins-hbase4:36177] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:51,144 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b37bc85591cde75b9b23b10dea6af810, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:51,145 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264031144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264031144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264031144"}]},"ts":"1676264031144"} 2023-02-13 04:53:51,147 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure b37bc85591cde75b9b23b10dea6af810, server=jenkins-hbase4.apache.org,40473,1676264030129}] 2023-02-13 04:53:51,301 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:51,301 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:51,302 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44070, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 04:53:51,306 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:51,306 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b37bc85591cde75b9b23b10dea6af810, NAME => 'hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:51,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:51,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,308 INFO [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,310 DEBUG [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info 2023-02-13 04:53:51,310 DEBUG [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info 2023-02-13 04:53:51,310 INFO [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b37bc85591cde75b9b23b10dea6af810 columnFamilyName info 2023-02-13 04:53:51,311 INFO [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] regionserver.HStore(310): Store=b37bc85591cde75b9b23b10dea6af810/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:51,312 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,312 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,315 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:51,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:51,319 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b37bc85591cde75b9b23b10dea6af810; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59240530, jitterRate=-0.1172473132610321}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:51,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b37bc85591cde75b9b23b10dea6af810: Running coprocessor pre-open hook at 1676264031307Writing region info on filesystem at 1676264031307Initializing all the Stores at 1676264031308 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264031308Cleaning up temporary data from old regions at 1676264031313 (+5 ms)Cleaning up detritus from prior splits at 1676264031313Running coprocessor post-open hooks at 1676264031319 (+6 ms)Region opened successfully at 1676264031319 2023-02-13 04:53:51,320 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810., pid=6, masterSystemTime=1676264031301 2023-02-13 04:53:51,324 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:51,324 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:51,325 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b37bc85591cde75b9b23b10dea6af810, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:51,326 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264031325"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264031325"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264031325"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264031325"}]},"ts":"1676264031325"} 2023-02-13 04:53:51,330 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 04:53:51,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure b37bc85591cde75b9b23b10dea6af810, server=jenkins-hbase4.apache.org,40473,1676264030129 in 181 msec 2023-02-13 04:53:51,333 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 04:53:51,333 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN in 343 msec 2023-02-13 04:53:51,334 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:51,334 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264031334"}]},"ts":"1676264031334"} 2023-02-13 04:53:51,336 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 04:53:51,338 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:51,340 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 409 msec 2023-02-13 04:53:51,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 04:53:51,434 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 04:53:51,434 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:51,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:51,438 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44084, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:51,440 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 04:53:51,450 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 04:53:51,458 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 17 msec 2023-02-13 04:53:51,462 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 04:53:51,470 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 04:53:51,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 11 msec 2023-02-13 04:53:51,489 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 04:53:51,491 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 04:53:51,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.311sec 2023-02-13 04:53:51,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 04:53:51,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 04:53:51,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 04:53:51,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36177,1676264030050-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 04:53:51,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36177,1676264030050-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 04:53:51,497 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 04:53:51,572 DEBUG [Listener at localhost/33627] zookeeper.ReadOnlyZKClient(139): Connect 0x7dffd1a0 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:51,578 DEBUG [Listener at localhost/33627] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71e47d0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:51,580 DEBUG [hconnection-0x109ae3a3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:51,582 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:51,584 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:51,586 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 04:53:51,586 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:51,587 INFO [Listener at localhost/33627] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 04:53:51,587 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 04:53:51,589 DEBUG [Listener at localhost/33627] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 04:53:51,591 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36198, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 04:53:51,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:51,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 04:53:51,597 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:51,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 04:53:51,598 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:51,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:51,602 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,603 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a empty. 2023-02-13 04:53:51,603 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,603 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 04:53:51,618 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:51,619 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => f0441c22e4864d797d7fc603dbec603a, NAME => 'restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp 2023-02-13 04:53:51,629 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:51,629 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing f0441c22e4864d797d7fc603dbec603a, disabling compactions & flushes 2023-02-13 04:53:51,630 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,630 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,630 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. after waiting 0 ms 2023-02-13 04:53:51,630 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,630 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,630 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for f0441c22e4864d797d7fc603dbec603a: Waiting for close lock at 1676264031629Disabling compacts and flushes for region at 1676264031629Disabling writes for close at 1676264031630 (+1 ms)Writing region close event to WAL at 1676264031630Closed at 1676264031630 2023-02-13 04:53:51,632 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:51,633 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264031633"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264031633"}]},"ts":"1676264031633"} 2023-02-13 04:53:51,635 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:51,636 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:51,636 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264031636"}]},"ts":"1676264031636"} 2023-02-13 04:53:51,638 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 04:53:51,642 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:51,642 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:51,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:51,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:51,643 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:51,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN}] 2023-02-13 04:53:51,645 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN 2023-02-13 04:53:51,646 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40715,1676264030105; forceNewPlan=false, retain=false 2023-02-13 04:53:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:51,796 INFO [jenkins-hbase4:36177] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:51,798 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f0441c22e4864d797d7fc603dbec603a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:51,798 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264031797"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264031797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264031797"}]},"ts":"1676264031797"} 2023-02-13 04:53:51,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f0441c22e4864d797d7fc603dbec603a, server=jenkins-hbase4.apache.org,40715,1676264030105}] 2023-02-13 04:53:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:51,954 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:51,954 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:51,956 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50416, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 04:53:51,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f0441c22e4864d797d7fc603dbec603a, NAME => 'restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:51,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:51,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,963 INFO [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,964 DEBUG [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/family 2023-02-13 04:53:51,964 DEBUG [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/family 2023-02-13 04:53:51,965 INFO [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f0441c22e4864d797d7fc603dbec603a columnFamilyName family 2023-02-13 04:53:51,965 INFO [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] regionserver.HStore(310): Store=f0441c22e4864d797d7fc603dbec603a/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:51,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:51,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:51,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f0441c22e4864d797d7fc603dbec603a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73777664, jitterRate=0.09937286376953125}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:51,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f0441c22e4864d797d7fc603dbec603a: Running coprocessor pre-open hook at 1676264031961Writing region info on filesystem at 1676264031961Initializing all the Stores at 1676264031962 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264031962Cleaning up temporary data from old regions at 1676264031967 (+5 ms)Cleaning up detritus from prior splits at 1676264031967Running coprocessor post-open hooks at 1676264031973 (+6 ms)Region opened successfully at 1676264031973 2023-02-13 04:53:51,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a., pid=11, masterSystemTime=1676264031954 2023-02-13 04:53:51,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:51,979 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f0441c22e4864d797d7fc603dbec603a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:51,979 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264031979"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264031979"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264031979"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264031979"}]},"ts":"1676264031979"} 2023-02-13 04:53:51,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 04:53:51,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f0441c22e4864d797d7fc603dbec603a, server=jenkins-hbase4.apache.org,40715,1676264030105 in 181 msec 2023-02-13 04:53:51,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 04:53:51,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN in 340 msec 2023-02-13 04:53:51,986 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:51,986 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264031986"}]},"ts":"1676264031986"} 2023-02-13 04:53:51,988 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 04:53:51,990 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:51,992 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 397 msec 2023-02-13 04:53:52,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 04:53:52,204 INFO [Listener at localhost/33627] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 04:53:52,205 DEBUG [Listener at localhost/33627] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 04:53:52,205 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:52,207 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 04:53:52,208 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:52,208 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 04:53:52,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:52,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 04:53:52,213 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:52,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 04:53:52,214 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:52,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:52,216 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,216 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 empty. 2023-02-13 04:53:52,217 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,217 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 04:53:52,234 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:52,235 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => bd5cf65a2c7621de28620d9e1b36b8d7, NAME => 'restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp 2023-02-13 04:53:52,252 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:52,253 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing bd5cf65a2c7621de28620d9e1b36b8d7, disabling compactions & flushes 2023-02-13 04:53:52,253 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,253 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,253 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. after waiting 0 ms 2023-02-13 04:53:52,253 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,253 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,253 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for bd5cf65a2c7621de28620d9e1b36b8d7: Waiting for close lock at 1676264032253Disabling compacts and flushes for region at 1676264032253Disabling writes for close at 1676264032253Writing region close event to WAL at 1676264032253Closed at 1676264032253 2023-02-13 04:53:52,255 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:52,256 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264032256"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264032256"}]},"ts":"1676264032256"} 2023-02-13 04:53:52,258 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:52,259 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:52,259 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264032259"}]},"ts":"1676264032259"} 2023-02-13 04:53:52,261 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 04:53:52,265 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:52,265 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:52,265 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:52,266 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:52,266 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:52,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN}] 2023-02-13 04:53:52,268 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN 2023-02-13 04:53:52,269 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40473,1676264030129; forceNewPlan=false, retain=false 2023-02-13 04:53:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:52,419 INFO [jenkins-hbase4:36177] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:52,420 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=bd5cf65a2c7621de28620d9e1b36b8d7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:52,420 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264032420"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264032420"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264032420"}]},"ts":"1676264032420"} 2023-02-13 04:53:52,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure bd5cf65a2c7621de28620d9e1b36b8d7, server=jenkins-hbase4.apache.org,40473,1676264030129}] 2023-02-13 04:53:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:52,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd5cf65a2c7621de28620d9e1b36b8d7, NAME => 'restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:52,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:52,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,581 INFO [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,583 DEBUG [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/family 2023-02-13 04:53:52,583 DEBUG [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/family 2023-02-13 04:53:52,583 INFO [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd5cf65a2c7621de28620d9e1b36b8d7 columnFamilyName family 2023-02-13 04:53:52,584 INFO [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] regionserver.HStore(310): Store=bd5cf65a2c7621de28620d9e1b36b8d7/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:52,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:52,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:52,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd5cf65a2c7621de28620d9e1b36b8d7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63630426, jitterRate=-0.051832765340805054}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:52,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd5cf65a2c7621de28620d9e1b36b8d7: Running coprocessor pre-open hook at 1676264032580Writing region info on filesystem at 1676264032580Initializing all the Stores at 1676264032581 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264032581Cleaning up temporary data from old regions at 1676264032585 (+4 ms)Cleaning up detritus from prior splits at 1676264032586 (+1 ms)Running coprocessor post-open hooks at 1676264032591 (+5 ms)Region opened successfully at 1676264032591 2023-02-13 04:53:52,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7., pid=14, masterSystemTime=1676264032574 2023-02-13 04:53:52,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:52,594 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=bd5cf65a2c7621de28620d9e1b36b8d7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:52,594 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264032594"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264032594"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264032594"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264032594"}]},"ts":"1676264032594"} 2023-02-13 04:53:52,598 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 04:53:52,598 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure bd5cf65a2c7621de28620d9e1b36b8d7, server=jenkins-hbase4.apache.org,40473,1676264030129 in 174 msec 2023-02-13 04:53:52,600 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 04:53:52,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN in 332 msec 2023-02-13 04:53:52,601 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:52,601 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264032601"}]},"ts":"1676264032601"} 2023-02-13 04:53:52,603 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 04:53:52,605 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:52,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 395 msec 2023-02-13 04:53:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 04:53:52,819 INFO [Listener at localhost/33627] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 04:53:52,819 DEBUG [Listener at localhost/33627] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 04:53:52,819 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:52,823 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 04:53:52,823 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:52,823 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 04:53:52,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 04:53:52,828 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 04:53:52,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 04:53:52,829 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 04:53:52,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:52,830 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:52,831 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 empty. 2023-02-13 04:53:52,832 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:52,832 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 04:53:52,847 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 04:53:52,848 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => 98f1b8d5338c83a011c1a33d5b920f87, NAME => 'restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.tmp 2023-02-13 04:53:52,860 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:52,860 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing 98f1b8d5338c83a011c1a33d5b920f87, disabling compactions & flushes 2023-02-13 04:53:52,860 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:52,860 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:52,860 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. after waiting 0 ms 2023-02-13 04:53:52,860 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:52,860 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:52,860 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for 98f1b8d5338c83a011c1a33d5b920f87: Waiting for close lock at 1676264032860Disabling compacts and flushes for region at 1676264032860Disabling writes for close at 1676264032860Writing region close event to WAL at 1676264032860Closed at 1676264032860 2023-02-13 04:53:52,863 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 04:53:52,864 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264032864"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264032864"}]},"ts":"1676264032864"} 2023-02-13 04:53:52,866 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 04:53:52,867 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 04:53:52,867 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264032867"}]},"ts":"1676264032867"} 2023-02-13 04:53:52,868 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 04:53:52,877 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:52,878 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:52,878 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:52,878 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:52,878 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:52,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN}] 2023-02-13 04:53:52,880 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN 2023-02-13 04:53:52,881 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40473,1676264030129; forceNewPlan=false, retain=false 2023-02-13 04:53:52,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:53,031 INFO [jenkins-hbase4:36177] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 04:53:53,032 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=98f1b8d5338c83a011c1a33d5b920f87, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:53,033 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264033032"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264033032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264033032"}]},"ts":"1676264033032"} 2023-02-13 04:53:53,035 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 98f1b8d5338c83a011c1a33d5b920f87, server=jenkins-hbase4.apache.org,40473,1676264030129}] 2023-02-13 04:53:53,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:53,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 98f1b8d5338c83a011c1a33d5b920f87, NAME => 'restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:53,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:53,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,194 INFO [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,196 DEBUG [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/family 2023-02-13 04:53:53,196 DEBUG [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/family 2023-02-13 04:53:53,196 INFO [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98f1b8d5338c83a011c1a33d5b920f87 columnFamilyName family 2023-02-13 04:53:53,197 INFO [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] regionserver.HStore(310): Store=98f1b8d5338c83a011c1a33d5b920f87/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:53,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 04:53:53,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 98f1b8d5338c83a011c1a33d5b920f87; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73290608, jitterRate=0.09211516380310059}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:53,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 98f1b8d5338c83a011c1a33d5b920f87: Running coprocessor pre-open hook at 1676264033193Writing region info on filesystem at 1676264033193Initializing all the Stores at 1676264033194 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264033194Cleaning up temporary data from old regions at 1676264033199 (+5 ms)Cleaning up detritus from prior splits at 1676264033199Running coprocessor post-open hooks at 1676264033204 (+5 ms)Region opened successfully at 1676264033204 2023-02-13 04:53:53,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87., pid=17, masterSystemTime=1676264033187 2023-02-13 04:53:53,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,207 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=98f1b8d5338c83a011c1a33d5b920f87, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:53,207 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264033207"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264033207"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264033207"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264033207"}]},"ts":"1676264033207"} 2023-02-13 04:53:53,211 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 04:53:53,211 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 98f1b8d5338c83a011c1a33d5b920f87, server=jenkins-hbase4.apache.org,40473,1676264030129 in 174 msec 2023-02-13 04:53:53,214 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 04:53:53,214 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN in 333 msec 2023-02-13 04:53:53,214 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 04:53:53,214 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264033214"}]},"ts":"1676264033214"} 2023-02-13 04:53:53,216 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 04:53:53,218 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 04:53:53,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 393 msec 2023-02-13 04:53:53,374 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 04:53:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36177] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 04:53:53,434 INFO [Listener at localhost/33627] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 04:53:53,434 DEBUG [Listener at localhost/33627] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 04:53:53,434 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:53,438 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 04:53:53,438 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:53,438 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 04:53:53,438 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:53,443 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:53,447 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:53,450 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:53,450 INFO [Listener at localhost/33627] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 04:53:53,453 INFO [Listener at localhost/33627] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 04:53:53,453 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(88): Shutting down HBase cluster 2023-02-13 04:53:53,453 INFO [Listener at localhost/33627] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:36177,5,FailOnTimeoutGroup] 2023-02-13 04:53:53,453 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36177,1676264030050' ***** 2023-02-13 04:53:53,453 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 04:53:53,454 DEBUG [M:0;jenkins-hbase4:36177] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c4a3242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:53,454 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36177,1676264030050 2023-02-13 04:53:53,455 DEBUG [M:0;jenkins-hbase4:36177] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bf83df4 to 127.0.0.1:59910 2023-02-13 04:53:53,455 DEBUG [M:0;jenkins-hbase4:36177] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,455 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36177,1676264030050; all regions closed. 2023-02-13 04:53:53,455 DEBUG [M:0;jenkins-hbase4:36177] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,455 DEBUG [M:0;jenkins-hbase4:36177] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 04:53:53,455 DEBUG [M:0;jenkins-hbase4:36177] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 04:53:53,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264030356] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264030356,5,FailOnTimeoutGroup] 2023-02-13 04:53:53,456 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 04:53:53,456 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:53,456 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:53,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264030358] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264030358,5,FailOnTimeoutGroup] 2023-02-13 04:53:53,457 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:53,457 DEBUG [Listener at localhost/33627] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 04:53:53,456 INFO [M:0;jenkins-hbase4:36177] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 04:53:53,458 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40715,1676264030105' ***** 2023-02-13 04:53:53,456 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:53,456 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:53,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:53,458 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:53,458 INFO [M:0;jenkins-hbase4:36177] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 04:53:53,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:53,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:53,459 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40473,1676264030129' ***** 2023-02-13 04:53:53,459 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:53,459 INFO [M:0;jenkins-hbase4:36177] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 04:53:53,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:53,460 INFO [RS:1;jenkins-hbase4:40473] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:53,459 INFO [RS:0;jenkins-hbase4:40715] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:53,460 INFO [RS:1;jenkins-hbase4:40473] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:53,460 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:53,460 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:53,460 DEBUG [M:0;jenkins-hbase4:36177] master.HMaster(1502): Stopping service threads 2023-02-13 04:53:53,459 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34837,1676264030153' ***** 2023-02-13 04:53:53,462 INFO [M:0;jenkins-hbase4:36177] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 04:53:53,460 INFO [RS:0;jenkins-hbase4:40715] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:53,462 INFO [M:0;jenkins-hbase4:36177] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 04:53:53,460 INFO [RS:1;jenkins-hbase4:40473] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:53,462 INFO [RS:0;jenkins-hbase4:40715] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:53,463 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(3304): Received CLOSE for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:53,463 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(3304): Received CLOSE for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:53,462 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:53,463 ERROR [M:0;jenkins-hbase4:36177] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 04:53:53,463 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(3304): Received CLOSE for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:53,464 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(3304): Received CLOSE for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:53,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b37bc85591cde75b9b23b10dea6af810, disabling compactions & flushes 2023-02-13 04:53:53,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f0441c22e4864d797d7fc603dbec603a, disabling compactions & flushes 2023-02-13 04:53:53,464 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:53,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:53,465 DEBUG [RS:0;jenkins-hbase4:40715] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a9c6b68 to 127.0.0.1:59910 2023-02-13 04:53:53,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:53,464 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:53,464 INFO [M:0;jenkins-hbase4:36177] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 04:53:53,464 INFO [RS:2;jenkins-hbase4:34837] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:53,465 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 04:53:53,465 INFO [RS:2;jenkins-hbase4:34837] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:53,466 DEBUG [RS:2;jenkins-hbase4:34837] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bc6ce21 to 127.0.0.1:59910 2023-02-13 04:53:53,466 DEBUG [RS:2;jenkins-hbase4:34837] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 04:53:53,465 DEBUG [RS:1;jenkins-hbase4:40473] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x521f8f41 to 127.0.0.1:59910 2023-02-13 04:53:53,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:53,466 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 04:53:53,465 DEBUG [RS:0;jenkins-hbase4:40715] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,466 DEBUG [M:0;jenkins-hbase4:36177] zookeeper.ZKUtil(398): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 04:53:53,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:53,467 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 04:53:53,467 WARN [M:0;jenkins-hbase4:36177] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 04:53:53,467 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 04:53:53,466 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 04:53:53,466 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-13 04:53:53,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. after waiting 0 ms 2023-02-13 04:53:53,468 DEBUG [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 04:53:53,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:53,466 DEBUG [RS:1;jenkins-hbase4:40473] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,465 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:53,468 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 04:53:53,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b37bc85591cde75b9b23b10dea6af810 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 04:53:53,468 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1479): Online Regions={f0441c22e4864d797d7fc603dbec603a=restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.} 2023-02-13 04:53:53,468 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 04:53:53,469 DEBUG [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1505): Waiting on f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:53,467 INFO [M:0;jenkins-hbase4:36177] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 04:53:53,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. after waiting 0 ms 2023-02-13 04:53:53,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 04:53:53,468 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1479): Online Regions={b37bc85591cde75b9b23b10dea6af810=hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810., 98f1b8d5338c83a011c1a33d5b920f87=restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87., bd5cf65a2c7621de28620d9e1b36b8d7=restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.} 2023-02-13 04:53:53,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 04:53:53,469 INFO [M:0;jenkins-hbase4:36177] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 04:53:53,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:53,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 04:53:53,470 DEBUG [M:0;jenkins-hbase4:36177] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 04:53:53,470 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:53,470 DEBUG [M:0;jenkins-hbase4:36177] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:53,470 DEBUG [M:0;jenkins-hbase4:36177] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 04:53:53,470 DEBUG [M:0;jenkins-hbase4:36177] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:53,470 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.23 KB heapSize=76.14 KB 2023-02-13 04:53:53,469 DEBUG [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1505): Waiting on 98f1b8d5338c83a011c1a33d5b920f87, b37bc85591cde75b9b23b10dea6af810, bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:53,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 04:53:53,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:53,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f0441c22e4864d797d7fc603dbec603a: Waiting for close lock at 1676264033464Running coprocessor pre-close hooks at 1676264033464Disabling compacts and flushes for region at 1676264033464Disabling writes for close at 1676264033469 (+5 ms)Writing region close event to WAL at 1676264033470 (+1 ms)Running coprocessor post-close hooks at 1676264033482 (+12 ms)Closed at 1676264033484 (+2 ms) 2023-02-13 04:53:53,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:53,502 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:53,516 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:53,517 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 04:53:53,518 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 04:53:53,519 INFO [M:0;jenkins-hbase4:36177] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.23 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/784fc8bb87b4420bbdf304eee6edabc3 2023-02-13 04:53:53,523 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/.tmp/info/6e48f975d08646189f9d4b7141f9e26a 2023-02-13 04:53:53,524 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 04:53:53,524 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 04:53:53,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/.tmp/info/c470f27a0d0948e99e0550e711f95a3e 2023-02-13 04:53:53,527 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:53,532 DEBUG [M:0;jenkins-hbase4:36177] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/784fc8bb87b4420bbdf304eee6edabc3 as hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/784fc8bb87b4420bbdf304eee6edabc3 2023-02-13 04:53:53,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/.tmp/info/c470f27a0d0948e99e0550e711f95a3e as hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info/c470f27a0d0948e99e0550e711f95a3e 2023-02-13 04:53:53,540 INFO [M:0;jenkins-hbase4:36177] regionserver.HStore(1080): Added hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/784fc8bb87b4420bbdf304eee6edabc3, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 04:53:53,540 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 04:53:53,540 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 04:53:53,541 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegion(2947): Finished flush of dataSize ~64.23 KB/65767, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 71ms, sequenceid=141, compaction requested=false 2023-02-13 04:53:53,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info/c470f27a0d0948e99e0550e711f95a3e, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 04:53:53,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for b37bc85591cde75b9b23b10dea6af810 in 79ms, sequenceid=6, compaction requested=false 2023-02-13 04:53:53,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 04:53:53,550 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:53,551 DEBUG [M:0;jenkins-hbase4:36177] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264033469Disabling compacts and flushes for region at 1676264033469Disabling writes for close at 1676264033470 (+1 ms)Obtaining lock to block concurrent updates at 1676264033470Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676264033470Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65767, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676264033471 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676264033472 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676264033473 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676264033478 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676264033478Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676264033533 (+55 ms)Finished flush of dataSize ~64.23 KB/65767, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 71ms, sequenceid=141, compaction requested=false at 1676264033541 (+8 ms)Writing region close event to WAL at 1676264033550 (+9 ms)Closed at 1676264033550 2023-02-13 04:53:53,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 04:53:53,562 INFO [M:0;jenkins-hbase4:36177] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 04:53:53,562 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:53,563 INFO [M:0;jenkins-hbase4:36177] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36177 2023-02-13 04:53:53,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:53,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b37bc85591cde75b9b23b10dea6af810: Waiting for close lock at 1676264033464Running coprocessor pre-close hooks at 1676264033464Disabling compacts and flushes for region at 1676264033464Disabling writes for close at 1676264033468 (+4 ms)Obtaining lock to block concurrent updates at 1676264033468Preparing flush snapshotting stores in b37bc85591cde75b9b23b10dea6af810 at 1676264033468Finished memstore snapshotting hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676264033469 (+1 ms)Flushing stores of hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. at 1676264033472 (+3 ms)Flushing b37bc85591cde75b9b23b10dea6af810/info: creating writer at 1676264033472Flushing b37bc85591cde75b9b23b10dea6af810/info: appending metadata at 1676264033476 (+4 ms)Flushing b37bc85591cde75b9b23b10dea6af810/info: closing flushed file at 1676264033476Flushing b37bc85591cde75b9b23b10dea6af810/info: reopening flushed file at 1676264033536 (+60 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for b37bc85591cde75b9b23b10dea6af810 in 79ms, sequenceid=6, compaction requested=false at 1676264033547 (+11 ms)Writing region close event to WAL at 1676264033550 (+3 ms)Running coprocessor post-close hooks at 1676264033564 (+14 ms)Closed at 1676264033565 (+1 ms) 2023-02-13 04:53:53,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:53,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 98f1b8d5338c83a011c1a33d5b920f87, disabling compactions & flushes 2023-02-13 04:53:53,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. after waiting 0 ms 2023-02-13 04:53:53,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,576 DEBUG [M:0;jenkins-hbase4:36177] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,36177,1676264030050 already deleted, retry=false 2023-02-13 04:53:53,577 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/.tmp/table/b3bd1248548744f98f034df72d27c82d 2023-02-13 04:53:53,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 04:53:53,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 98f1b8d5338c83a011c1a33d5b920f87: Waiting for close lock at 1676264033565Running coprocessor pre-close hooks at 1676264033565Disabling compacts and flushes for region at 1676264033565Disabling writes for close at 1676264033566 (+1 ms)Writing region close event to WAL at 1676264033573 (+7 ms)Running coprocessor post-close hooks at 1676264033579 (+6 ms)Closed at 1676264033579 2023-02-13 04:53:53,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:53,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd5cf65a2c7621de28620d9e1b36b8d7, disabling compactions & flushes 2023-02-13 04:53:53,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:53,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:53,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. after waiting 0 ms 2023-02-13 04:53:53,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:53,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 04:53:53,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:53,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd5cf65a2c7621de28620d9e1b36b8d7: Waiting for close lock at 1676264033580Running coprocessor pre-close hooks at 1676264033580Disabling compacts and flushes for region at 1676264033580Disabling writes for close at 1676264033580Writing region close event to WAL at 1676264033581 (+1 ms)Running coprocessor post-close hooks at 1676264033585 (+4 ms)Closed at 1676264033585 2023-02-13 04:53:53,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:53,586 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/.tmp/info/6e48f975d08646189f9d4b7141f9e26a as hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info/6e48f975d08646189f9d4b7141f9e26a 2023-02-13 04:53:53,592 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info/6e48f975d08646189f9d4b7141f9e26a, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 04:53:53,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/.tmp/table/b3bd1248548744f98f034df72d27c82d as hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table/b3bd1248548744f98f034df72d27c82d 2023-02-13 04:53:53,599 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table/b3bd1248548744f98f034df72d27c82d, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 04:53:53,600 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 131ms, sequenceid=24, compaction requested=false 2023-02-13 04:53:53,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 04:53:53,606 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 04:53:53,607 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 04:53:53,607 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:53,607 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264033467Running coprocessor pre-close hooks at 1676264033467Disabling compacts and flushes for region at 1676264033467Disabling writes for close at 1676264033469 (+2 ms)Obtaining lock to block concurrent updates at 1676264033470 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676264033470Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676264033470Flushing stores of hbase:meta,,1.1588230740 at 1676264033472 (+2 ms)Flushing 1588230740/info: creating writer at 1676264033472Flushing 1588230740/info: appending metadata at 1676264033477 (+5 ms)Flushing 1588230740/info: closing flushed file at 1676264033477Flushing 1588230740/table: creating writer at 1676264033532 (+55 ms)Flushing 1588230740/table: appending metadata at 1676264033548 (+16 ms)Flushing 1588230740/table: closing flushed file at 1676264033548Flushing 1588230740/info: reopening flushed file at 1676264033586 (+38 ms)Flushing 1588230740/table: reopening flushed file at 1676264033593 (+7 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 131ms, sequenceid=24, compaction requested=false at 1676264033600 (+7 ms)Writing region close event to WAL at 1676264033603 (+3 ms)Running coprocessor post-close hooks at 1676264033607 (+4 ms)Closed at 1676264033607 2023-02-13 04:53:53,608 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:53,668 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34837,1676264030153; all regions closed. 2023-02-13 04:53:53,669 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40715,1676264030105; all regions closed. 2023-02-13 04:53:53,671 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40473,1676264030129; all regions closed. 2023-02-13 04:53:53,683 DEBUG [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:53,683 INFO [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34837%2C1676264030153.meta:.meta(num 1676264030854) 2023-02-13 04:53:53,685 DEBUG [RS:0;jenkins-hbase4:40715] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:53,685 INFO [RS:0;jenkins-hbase4:40715] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40715%2C1676264030105:(num 1676264030664) 2023-02-13 04:53:53,685 DEBUG [RS:0;jenkins-hbase4:40715] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,685 INFO [RS:0;jenkins-hbase4:40715] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:53,687 INFO [RS:0;jenkins-hbase4:40715] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:53,687 INFO [RS:0;jenkins-hbase4:40715] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:53,687 INFO [RS:0;jenkins-hbase4:40715] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:53,687 INFO [RS:0;jenkins-hbase4:40715] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:53,687 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:53,688 DEBUG [RS:1;jenkins-hbase4:40473] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:53,688 INFO [RS:1;jenkins-hbase4:40473] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40473%2C1676264030129:(num 1676264030646) 2023-02-13 04:53:53,688 DEBUG [RS:1;jenkins-hbase4:40473] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,688 INFO [RS:1;jenkins-hbase4:40473] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:53,689 INFO [RS:1;jenkins-hbase4:40473] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:53,689 INFO [RS:1;jenkins-hbase4:40473] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:53,689 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:53,689 INFO [RS:1;jenkins-hbase4:40473] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:53,689 INFO [RS:1;jenkins-hbase4:40473] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:53,690 INFO [RS:0;jenkins-hbase4:40715] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40715 2023-02-13 04:53:53,690 INFO [RS:1;jenkins-hbase4:40473] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40473 2023-02-13 04:53:53,693 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:53,694 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:53,694 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:53,694 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:53,694 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@69ffea0 rejected from java.util.concurrent.ThreadPoolExecutor@2a06aab[Shutting down, pool size = 1, active threads = 0, queued tasks = 1, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,694 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:53,694 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:53,696 DEBUG [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:53,696 INFO [RS:2;jenkins-hbase4:34837] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34837%2C1676264030153:(num 1676264030650) 2023-02-13 04:53:53,696 DEBUG [RS:2;jenkins-hbase4:34837] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:53,696 INFO [RS:2;jenkins-hbase4:34837] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:53,696 INFO [RS:2;jenkins-hbase4:34837] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:53,696 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:53,697 INFO [RS:2;jenkins-hbase4:34837] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34837 2023-02-13 04:53:53,697 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:53,698 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:53,698 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:53,698 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f3c636d rejected from java.util.concurrent.ThreadPoolExecutor@2a06aab[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,699 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:53,699 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264030153 2023-02-13 04:53:53,699 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61de38d9 rejected from java.util.concurrent.ThreadPoolExecutor@65e1a03a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,699 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@29f4f482 rejected from java.util.concurrent.ThreadPoolExecutor@2dcb7564[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,859 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:53,859 INFO [RS:2;jenkins-hbase4:34837] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34837,1676264030153; zookeeper connection closed. 2023-02-13 04:53:53,859 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@101fe2f rejected from java.util.concurrent.ThreadPoolExecutor@65e1a03a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,860 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c0729060003, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:53,860 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7901844d rejected from java.util.concurrent.ThreadPoolExecutor@65e1a03a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,860 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5a37795a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5a37795a 2023-02-13 04:53:53,959 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:53,959 INFO [RS:1;jenkins-hbase4:40473] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40473,1676264030129; zookeeper connection closed. 2023-02-13 04:53:53,960 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@57358e0a rejected from java.util.concurrent.ThreadPoolExecutor@2dcb7564[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:53,960 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c0729060002, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:53,960 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@16399c97] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@16399c97 2023-02-13 04:53:53,960 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ffd1e86 rejected from java.util.concurrent.ThreadPoolExecutor@2dcb7564[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:54,060 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:54,060 INFO [RS:0;jenkins-hbase4:40715] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40715,1676264030105; zookeeper connection closed. 2023-02-13 04:53:54,060 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@775b3f88 rejected from java.util.concurrent.ThreadPoolExecutor@2a06aab[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:54,060 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c0729060001, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:54,060 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4b462343 rejected from java.util.concurrent.ThreadPoolExecutor@2a06aab[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:54,060 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@426a03c0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@426a03c0 2023-02-13 04:53:54,062 INFO [Listener at localhost/33627] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 04:53:54,160 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:54,160 INFO [M:0;jenkins-hbase4:36177] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36177,1676264030050; zookeeper connection closed. 2023-02-13 04:53:54,160 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@447f9b83 rejected from java.util.concurrent.ThreadPoolExecutor@404017d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:54,161 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(93): Sleeping a bit 2023-02-13 04:53:54,161 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:36177-0x1004c0729060000, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:54,161 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c1690e8 rejected from java.util.concurrent.ThreadPoolExecutor@404017d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:56,161 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(96): Starting cluster the second time with the same ports 2023-02-13 04:53:56,174 INFO [Listener at localhost/33627] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 04:53:56,174 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,174 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,174 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:56,174 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,175 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:56,175 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:56,175 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33465 2023-02-13 04:53:56,176 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,177 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,178 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33465 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:56,181 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:334650x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:56,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:33465-0x1004c0729060009 connected 2023-02-13 04:53:56,185 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:56,185 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 04:53:56,186 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:56,186 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33465 2023-02-13 04:53:56,187 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33465 2023-02-13 04:53:56,187 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33465 2023-02-13 04:53:56,187 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33465 2023-02-13 04:53:56,187 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33465 2023-02-13 04:53:56,187 INFO [Listener at localhost/33627] master.HMaster(439): hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec, hbase.cluster.distributed=false 2023-02-13 04:53:56,188 DEBUG [pool-500-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 04:53:56,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,190 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:56,191 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:56,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:56,194 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:56,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,33465,1676264036173 from backup master directory 2023-02-13 04:53:56,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:56,195 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:56,195 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,196 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 04:53:56,196 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:56,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,204 INFO [Listener at localhost/33627] client.ConnectionUtils(127): regionserver/jenkins-hbase4:40715 server-side Connection retries=45 2023-02-13 04:53:56,204 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,204 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,204 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:56,204 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,205 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:56,205 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:56,205 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40715 2023-02-13 04:53:56,206 INFO [Listener at localhost/33627] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:56,208 DEBUG [Listener at localhost/33627] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:56,208 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,209 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,210 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40715 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:56,214 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:407150x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:56,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40715-0x1004c072906000a connected 2023-02-13 04:53:56,215 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:56,216 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 04:53:56,217 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:56,223 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40715 2023-02-13 04:53:56,224 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40715 2023-02-13 04:53:56,224 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40715 2023-02-13 04:53:56,225 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40715 2023-02-13 04:53:56,226 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40715 2023-02-13 04:53:56,228 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:56,228 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:56,230 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:56,230 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:56,238 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:56,239 DEBUG [RS:3;jenkins-hbase4:40715] zookeeper.ReadOnlyZKClient(139): Connect 0x665c3c71 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:56,244 DEBUG [RS:3;jenkins-hbase4:40715] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@689bdeca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:56,245 DEBUG [RS:3;jenkins-hbase4:40715] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ca45701, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:56,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7785a270 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:56,256 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:40715 2023-02-13 04:53:56,256 INFO [RS:3;jenkins-hbase4:40715] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:56,256 INFO [RS:3;jenkins-hbase4:40715] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:56,256 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:56,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34a54902, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:56,256 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33465,1676264036173 with isa=jenkins-hbase4.apache.org/172.31.14.131:40715, startcode=1676264036203 2023-02-13 04:53:56,256 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 04:53:56,256 DEBUG [RS:3;jenkins-hbase4:40715] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:56,257 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 04:53:56,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:56,259 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53859, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.9 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:56,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:56,261 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 04:53:56,261 WARN [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 04:53:56,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050 to hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050-dead as it is dead 2023-02-13 04:53:56,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050-dead/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 2023-02-13 04:53:56,265 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050-dead/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 after 1ms 2023-02-13 04:53:56,265 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050-dead/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 to hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 2023-02-13 04:53:56,265 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,36177,1676264030050-dead 2023-02-13 04:53:56,266 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33465%2C1676264036173, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,33465,1676264036173, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/oldWALs, maxLogs=10 2023-02-13 04:53:56,282 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:56,282 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:56,282 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:56,284 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/WALs/jenkins-hbase4.apache.org,33465,1676264036173/jenkins-hbase4.apache.org%2C33465%2C1676264036173.1676264036268 2023-02-13 04:53:56,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK]] 2023-02-13 04:53:56,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:56,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:56,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:56,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:56,286 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:56,287 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 04:53:56,288 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 04:53:56,294 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/784fc8bb87b4420bbdf304eee6edabc3 2023-02-13 04:53:56,294 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:56,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 04:53:56,295 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 2023-02-13 04:53:56,302 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 2023-02-13 04:53:56,302 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C36177%2C1676264030050.1676264030257 2023-02-13 04:53:56,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 04:53:56,309 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 04:53:56,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59726369, jitterRate=-0.11000774800777435}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:56,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676264036285Initializing all the Stores at 1676264036286 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264036286Cleaning up temporary data from old regions at 1676264036303 (+17 ms)Cleaning up detritus from prior splits at 1676264036303Region opened successfully at 1676264036310 (+7 ms) 2023-02-13 04:53:56,310 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 04:53:56,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 04:53:56,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 04:53:56,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 04:53:56,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 04:53:56,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 04:53:56,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 04:53:56,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 04:53:56,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 04:53:56,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 04:53:56,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 04:53:56,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 04:53:56,317 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-13 04:53:56,317 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 04:53:56,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 04:53:56,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,34837,1676264030153, table=hbase:meta, region=1588230740 2023-02-13 04:53:56,319 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 04:53:56,322 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40473,1676264030129 already deleted, retry=false 2023-02-13 04:53:56,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,40473,1676264030129 on jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,323 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,40473,1676264030129, splitWal=true, meta=false 2023-02-13 04:53:56,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,40473,1676264030129 (carryingMeta=false) jenkins-hbase4.apache.org,40473,1676264030129/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@4106fc4a[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 04:53:56,325 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40715,1676264030105 already deleted, retry=false 2023-02-13 04:53:56,325 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,40715,1676264030105 on jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,325 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,40715,1676264030105, splitWal=true, meta=false 2023-02-13 04:53:56,326 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,40715,1676264030105 (carryingMeta=false) jenkins-hbase4.apache.org,40715,1676264030105/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1d28b4be[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 04:53:56,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34837,1676264030153 already deleted, retry=false 2023-02-13 04:53:56,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,34837,1676264030153 on jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,327 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,34837,1676264030153, splitWal=true, meta=true 2023-02-13 04:53:56,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=20 for jenkins-hbase4.apache.org,34837,1676264030153 (carryingMeta=true) jenkins-hbase4.apache.org,34837,1676264030153/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@789efcd0[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 04:53:56,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 04:53:56,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 04:53:56,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 04:53:56,338 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 04:53:56,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 04:53:56,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 04:53:56,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 04:53:56,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 04:53:56,340 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,33465,1676264036173, sessionid=0x1004c0729060009, setting cluster-up flag (Was=true) 2023-02-13 04:53:56,342 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 04:53:56,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 04:53:56,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:56,347 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/.hbase-snapshot/.tmp 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:56,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676264066350 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 04:53:56,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 04:53:56,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 04:53:56,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 04:53:56,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 04:53:56,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 04:53:56,353 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264036353,5,FailOnTimeoutGroup] 2023-02-13 04:53:56,354 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264036353,5,FailOnTimeoutGroup] 2023-02-13 04:53:56,354 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34837,1676264030153; numProcessing=1 2023-02-13 04:53:56,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,354 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=20, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34837,1676264030153, splitWal=true, meta=true 2023-02-13 04:53:56,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 04:53:56,354 DEBUG [PEWorker-2] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40473,1676264030129; numProcessing=2 2023-02-13 04:53:56,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,354 DEBUG [PEWorker-4] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40715,1676264030105; numProcessing=3 2023-02-13 04:53:56,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,354 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,40715,1676264030105, splitWal=true, meta=false 2023-02-13 04:53:56,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676264036191, completionTime=-1 2023-02-13 04:53:56,354 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,40473,1676264030129, splitWal=true, meta=false 2023-02-13 04:53:56,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=0; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 04:53:56,356 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34837,1676264030153, splitWal=true, meta=true, isMeta: true 2023-02-13 04:53:56,358 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153-splitting 2023-02-13 04:53:56,359 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153-splitting dir is empty, no logs to split. 2023-02-13 04:53:56,359 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,34837,1676264030153 WAL count=0, meta=true 2023-02-13 04:53:56,361 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33465,1676264036173 with isa=jenkins-hbase4.apache.org/172.31.14.131:40715, startcode=1676264036203 2023-02-13 04:53:56,362 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153-splitting dir is empty, no logs to split. 2023-02-13 04:53:56,362 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,34837,1676264030153 WAL count=0, meta=true 2023-02-13 04:53:56,362 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,34837,1676264030153 WAL splitting is done? wals=0, meta=true 2023-02-13 04:53:56,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,363 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:56,363 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44887 2023-02-13 04:53:56,363 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:56,364 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 04:53:56,364 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:56,365 DEBUG [RS:3;jenkins-hbase4:40715] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,365 WARN [RS:3;jenkins-hbase4:40715] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:56,365 INFO [RS:3;jenkins-hbase4:40715] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:56,365 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,368 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 04:53:56,369 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40715,1676264036203] 2023-02-13 04:53:56,370 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,34837,1676264030153; forceNewPlan=false, retain=false 2023-02-13 04:53:56,375 DEBUG [RS:3;jenkins-hbase4:40715] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,376 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:56,376 INFO [RS:3;jenkins-hbase4:40715] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:56,379 INFO [RS:3;jenkins-hbase4:40715] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:56,379 INFO [RS:3;jenkins-hbase4:40715] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:56,379 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,379 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:56,380 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:40715, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,381 DEBUG [RS:3;jenkins-hbase4:40715] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:40715, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,384 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,385 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,385 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,395 INFO [RS:3;jenkins-hbase4:40715] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:56,395 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40715,1676264036203-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,405 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=1; waited=50ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 04:53:56,406 INFO [RS:3;jenkins-hbase4:40715] regionserver.Replication(203): jenkins-hbase4.apache.org,40715,1676264036203 started 2023-02-13 04:53:56,406 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40715,1676264036203, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40715, sessionid=0x1004c072906000a 2023-02-13 04:53:56,406 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:56,406 DEBUG [RS:3;jenkins-hbase4:40715] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,406 DEBUG [RS:3;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40715,1676264036203' 2023-02-13 04:53:56,406 DEBUG [RS:3;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:56,406 DEBUG [RS:3;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40715,1676264036203' 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:56,407 DEBUG [RS:3;jenkins-hbase4:40715] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:56,407 INFO [RS:3;jenkins-hbase4:40715] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:56,407 INFO [RS:3;jenkins-hbase4:40715] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:56,416 INFO [Listener at localhost/33627] client.ConnectionUtils(127): regionserver/jenkins-hbase4:40473 server-side Connection retries=45 2023-02-13 04:53:56,416 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,416 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,417 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:56,417 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,417 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:56,417 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:56,417 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40473 2023-02-13 04:53:56,418 INFO [Listener at localhost/33627] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:56,419 DEBUG [Listener at localhost/33627] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:56,419 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,421 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,421 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40473 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:56,424 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:404730x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:56,425 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): regionserver:404730x0, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:56,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40473-0x1004c072906000d connected 2023-02-13 04:53:56,426 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 04:53:56,427 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:56,427 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40473 2023-02-13 04:53:56,428 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40473 2023-02-13 04:53:56,428 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40473 2023-02-13 04:53:56,428 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40473 2023-02-13 04:53:56,428 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40473 2023-02-13 04:53:56,430 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:56,430 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:56,432 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:56,432 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:56,435 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:56,435 DEBUG [RS:4;jenkins-hbase4:40473] zookeeper.ReadOnlyZKClient(139): Connect 0x01af5c95 to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:56,440 DEBUG [RS:4;jenkins-hbase4:40473] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f68a2d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:56,441 DEBUG [RS:4;jenkins-hbase4:40473] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b5737e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:56,451 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:4;jenkins-hbase4:40473 2023-02-13 04:53:56,451 INFO [RS:4;jenkins-hbase4:40473] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:56,451 INFO [RS:4;jenkins-hbase4:40473] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:56,451 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:56,452 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33465,1676264036173 with isa=jenkins-hbase4.apache.org/172.31.14.131:40473, startcode=1676264036416 2023-02-13 04:53:56,452 DEBUG [RS:4;jenkins-hbase4:40473] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:56,454 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43481, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.10 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:56,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,455 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:56,455 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44887 2023-02-13 04:53:56,455 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:56,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=100ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 04:53:56,456 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:56,456 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:56,456 DEBUG [RS:4;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,456 WARN [RS:4;jenkins-hbase4:40473] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:56,457 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40473,1676264036416] 2023-02-13 04:53:56,457 INFO [RS:4;jenkins-hbase4:40473] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:56,457 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,461 DEBUG [RS:4;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,461 DEBUG [RS:4;jenkins-hbase4:40473] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,462 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:56,462 INFO [RS:4;jenkins-hbase4:40473] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:56,463 INFO [RS:4;jenkins-hbase4:40473] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:56,463 INFO [RS:4;jenkins-hbase4:40473] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:56,464 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,464 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:56,465 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:40473, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:56,465 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,466 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,466 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,466 DEBUG [RS:4;jenkins-hbase4:40473] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:40473, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,467 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,467 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,467 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,478 INFO [RS:4;jenkins-hbase4:40473] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:56,478 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40473,1676264036416-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,488 INFO [RS:4;jenkins-hbase4:40473] regionserver.Replication(203): jenkins-hbase4.apache.org,40473,1676264036416 started 2023-02-13 04:53:56,488 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40473,1676264036416, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40473, sessionid=0x1004c072906000d 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40473,1676264036416' 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40473,1676264036416' 2023-02-13 04:53:56,489 DEBUG [RS:4;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:56,490 DEBUG [RS:4;jenkins-hbase4:40473] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:56,490 DEBUG [RS:4;jenkins-hbase4:40473] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:56,490 INFO [RS:4;jenkins-hbase4:40473] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:56,490 INFO [RS:4;jenkins-hbase4:40473] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:56,499 INFO [Listener at localhost/33627] client.ConnectionUtils(127): regionserver/jenkins-hbase4:34837 server-side Connection retries=45 2023-02-13 04:53:56,499 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,499 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,499 INFO [Listener at localhost/33627] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 04:53:56,499 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 04:53:56,499 INFO [Listener at localhost/33627] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 04:53:56,500 INFO [Listener at localhost/33627] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 04:53:56,500 INFO [Listener at localhost/33627] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34837 2023-02-13 04:53:56,501 INFO [Listener at localhost/33627] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 04:53:56,502 DEBUG [Listener at localhost/33627] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 04:53:56,502 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,503 INFO [Listener at localhost/33627] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 04:53:56,504 INFO [Listener at localhost/33627] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34837 connecting to ZooKeeper ensemble=127.0.0.1:59910 2023-02-13 04:53:56,507 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:348370x0, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 04:53:56,508 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): regionserver:348370x0, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 04:53:56,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34837-0x1004c072906000f connected 2023-02-13 04:53:56,510 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(162): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 04:53:56,510 DEBUG [Listener at localhost/33627] zookeeper.ZKUtil(164): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 04:53:56,511 INFO [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40715%2C1676264036203, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264036203, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:56,511 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34837 2023-02-13 04:53:56,511 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34837 2023-02-13 04:53:56,511 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34837 2023-02-13 04:53:56,514 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34837 2023-02-13 04:53:56,515 DEBUG [Listener at localhost/33627] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34837 2023-02-13 04:53:56,516 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(952): ClusterId : 7a2bcff8-f317-4fa2-bed6-6250b2883fea 2023-02-13 04:53:56,516 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 04:53:56,519 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 04:53:56,519 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 04:53:56,520 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 04:53:56,523 DEBUG [RS:5;jenkins-hbase4:34837] zookeeper.ReadOnlyZKClient(139): Connect 0x5acb77fc to 127.0.0.1:59910 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 04:53:56,523 DEBUG [jenkins-hbase4:33465] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 04:53:56,528 DEBUG [RS:5;jenkins-hbase4:34837] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@436bdcad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 04:53:56,528 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:56,528 DEBUG [RS:5;jenkins-hbase4:34837] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bdb8942, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:56,529 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:56,529 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:56,531 DEBUG [jenkins-hbase4:33465] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:53:56,531 DEBUG [jenkins-hbase4:33465] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 04:53:56,531 DEBUG [jenkins-hbase4:33465] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 04:53:56,531 DEBUG [jenkins-hbase4:33465] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 04:53:56,533 INFO [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264036203/jenkins-hbase4.apache.org%2C40715%2C1676264036203.1676264036511 2023-02-13 04:53:56,533 INFO [jenkins-hbase4:33465] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 0 retained the pre-restart assignment. 1 regions were assigned to random hosts, since the old hosts for these regions are no longer present in the cluster. These hosts were: 2023-02-13 04:53:56,534 DEBUG [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK], DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK]] 2023-02-13 04:53:56,534 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40715,1676264036203, state=OPENING 2023-02-13 04:53:56,535 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 04:53:56,535 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 04:53:56,535 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40715,1676264036203}] 2023-02-13 04:53:56,544 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:5;jenkins-hbase4:34837 2023-02-13 04:53:56,544 INFO [RS:5;jenkins-hbase4:34837] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 04:53:56,544 INFO [RS:5;jenkins-hbase4:34837] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 04:53:56,544 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 04:53:56,545 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33465,1676264036173 with isa=jenkins-hbase4.apache.org/172.31.14.131:34837, startcode=1676264036498 2023-02-13 04:53:56,545 DEBUG [RS:5;jenkins-hbase4:34837] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 04:53:56,547 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54147, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.11 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 04:53:56,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,547 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec 2023-02-13 04:53:56,547 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44887 2023-02-13 04:53:56,547 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 04:53:56,550 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:56,550 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:56,550 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:56,551 DEBUG [RS:5;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,551 WARN [RS:5;jenkins-hbase4:34837] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 04:53:56,551 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34837,1676264036498] 2023-02-13 04:53:56,551 INFO [RS:5;jenkins-hbase4:34837] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:56,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,551 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=200ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 04:53:56,557 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPENING, ts=1676264036535, server=jenkins-hbase4.apache.org,40715,1676264036203}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 04:53:56,558 DEBUG [RS:5;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,558 DEBUG [RS:5;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,559 DEBUG [RS:5;jenkins-hbase4:34837] zookeeper.ZKUtil(162): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:56,559 DEBUG [RS:5;jenkins-hbase4:34837] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 04:53:56,560 INFO [RS:5;jenkins-hbase4:34837] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 04:53:56,561 INFO [RS:5;jenkins-hbase4:34837] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 04:53:56,562 INFO [RS:5;jenkins-hbase4:34837] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 04:53:56,562 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,562 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 04:53:56,563 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:34837, corePoolSize=2, maxPoolSize=2 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,564 DEBUG [RS:5;jenkins-hbase4:34837] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:34837, corePoolSize=1, maxPoolSize=1 2023-02-13 04:53:56,567 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,567 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,567 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,583 INFO [RS:5;jenkins-hbase4:34837] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 04:53:56,584 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34837,1676264036498-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:56,593 INFO [RS:4;jenkins-hbase4:40473] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40473%2C1676264036416, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264036416, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:56,600 INFO [RS:5;jenkins-hbase4:34837] regionserver.Replication(203): jenkins-hbase4.apache.org,34837,1676264036498 started 2023-02-13 04:53:56,600 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34837,1676264036498, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34837, sessionid=0x1004c072906000f 2023-02-13 04:53:56,600 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 04:53:56,600 DEBUG [RS:5;jenkins-hbase4:34837] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,600 DEBUG [RS:5;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34837,1676264036498' 2023-02-13 04:53:56,600 DEBUG [RS:5;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 04:53:56,600 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:56,601 DEBUG [RS:5;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 04:53:56,601 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 04:53:56,601 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 04:53:56,601 DEBUG [RS:5;jenkins-hbase4:34837] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,601 DEBUG [RS:5;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34837,1676264036498' 2023-02-13 04:53:56,601 DEBUG [RS:5;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 04:53:56,602 DEBUG [RS:5;jenkins-hbase4:34837] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 04:53:56,602 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60992, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:56,602 DEBUG [RS:5;jenkins-hbase4:34837] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 04:53:56,602 INFO [RS:5;jenkins-hbase4:34837] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 04:53:56,602 INFO [RS:5;jenkins-hbase4:34837] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 04:53:56,603 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34837] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:60992 deadline: 1676264096603, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:56,613 DEBUG [RS-EventLoopGroup-15-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:56,614 DEBUG [RS-EventLoopGroup-15-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:56,622 DEBUG [RS-EventLoopGroup-15-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:56,624 INFO [RS:4;jenkins-hbase4:40473] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264036416/jenkins-hbase4.apache.org%2C40473%2C1676264036416.1676264036594 2023-02-13 04:53:56,625 DEBUG [RS:4;jenkins-hbase4:40473] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK]] 2023-02-13 04:53:56,693 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,693 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:56,694 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42078, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 04:53:56,698 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 04:53:56,698 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 04:53:56,700 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40715%2C1676264036203.meta, suffix=.meta, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264036203, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:56,704 INFO [RS:5;jenkins-hbase4:34837] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34837%2C1676264036498, suffix=, logDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264036498, archiveDir=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs, maxLogs=32 2023-02-13 04:53:56,706 WARN [ReadOnlyZKClient-127.0.0.1:59910@0x7dffd1a0] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-02-13 04:53:56,706 DEBUG [Listener at localhost/33627] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:56,708 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42084, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:56,709 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40715] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:42084 deadline: 1676264096708, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:56,715 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:56,715 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:56,715 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:56,718 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264036203/jenkins-hbase4.apache.org%2C40715%2C1676264036203.meta.1676264036700.meta 2023-02-13 04:53:56,718 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK]] 2023-02-13 04:53:56,718 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:56,720 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 04:53:56,720 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK] 2023-02-13 04:53:56,720 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 04:53:56,720 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 04:53:56,720 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 04:53:56,720 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK] 2023-02-13 04:53:56,720 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:56,721 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 04:53:56,721 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 04:53:56,721 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK] 2023-02-13 04:53:56,724 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 04:53:56,724 INFO [RS:5;jenkins-hbase4:34837] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264036498/jenkins-hbase4.apache.org%2C34837%2C1676264036498.1676264036704 2023-02-13 04:53:56,725 DEBUG [RS:5;jenkins-hbase4:34837] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43657,DS-2db251de-67f9-46dc-9c0a-39f27904fc17,DISK], DatanodeInfoWithStorage[127.0.0.1:39251,DS-adb1b923-3f68-4792-94e0-90ea0aea59c2,DISK], DatanodeInfoWithStorage[127.0.0.1:39881,DS-6f049793-407a-4f8d-aa70-9720627cd3e0,DISK]] 2023-02-13 04:53:56,725 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info 2023-02-13 04:53:56,725 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info 2023-02-13 04:53:56,725 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 04:53:56,732 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info/6e48f975d08646189f9d4b7141f9e26a 2023-02-13 04:53:56,732 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:56,732 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 04:53:56,733 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:56,733 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/rep_barrier 2023-02-13 04:53:56,733 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 04:53:56,734 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:56,734 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 04:53:56,735 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table 2023-02-13 04:53:56,735 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table 2023-02-13 04:53:56,735 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 04:53:56,741 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/table/b3bd1248548744f98f034df72d27c82d 2023-02-13 04:53:56,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:56,742 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740 2023-02-13 04:53:56,743 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740 2023-02-13 04:53:56,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 04:53:56,747 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 04:53:56,748 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65089328, jitterRate=-0.03009343147277832}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 04:53:56,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676264036721Writing region info on filesystem at 1676264036721Initializing all the Stores at 1676264036722 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264036722Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264036724 (+2 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264036724Cleaning up temporary data from old regions at 1676264036744 (+20 ms)Cleaning up detritus from prior splits at 1676264036745 (+1 ms)Running coprocessor post-open hooks at 1676264036748 (+3 ms)Region opened successfully at 1676264036748 2023-02-13 04:53:56,749 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=22, masterSystemTime=1676264036693 2023-02-13 04:53:56,754 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 04:53:56,754 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:40715-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 04:53:56,755 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40715,1676264036203, state=OPEN 2023-02-13 04:53:56,756 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 04:53:56,757 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 04:53:56,759 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-13 04:53:56,759 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40715,1676264036203 in 222 msec 2023-02-13 04:53:56,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-02-13 04:53:56,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 395 msec 2023-02-13 04:53:56,925 WARN [RPCClient-NioEventLoopGroup-6-6] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:36177 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36177 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:56,925 WARN [Listener at localhost/33627] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:36177 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36177 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36177 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 04:53:56,925 DEBUG [RPCClient-NioEventLoopGroup-6-6] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:36177 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36177 2023-02-13 04:53:56,926 DEBUG [Listener at localhost/33627] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:36177 this server is in the failed servers list 2023-02-13 04:53:56,927 WARN [Listener at localhost/33627] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:36177 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:36177 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnClusterRestart(TestRetainAssignmentOnRestart.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:36177 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 04:53:56,928 DEBUG [Listener at localhost/33627] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 04:53:56,931 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59748, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 04:53:56,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:56,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.CallRunner(144): callId: 50 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:59748 deadline: 1676264096931, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 04:53:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.CallRunner(144): callId: 52 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:59748 deadline: 1676264097033, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 04:53:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.CallRunner(144): callId: 54 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:59748 deadline: 1676264097236, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 04:53:57,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:57,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.CallRunner(144): callId: 56 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:59748 deadline: 1676264097540, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 04:53:57,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 04:53:57,560 DEBUG [hconnection-0x5a2d5f62-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:57,562 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-13 04:53:57,565 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=b37bc85591cde75b9b23b10dea6af810, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40473,1676264030129, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129, openSeqNum=2 2023-02-13 04:53:57,565 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=f0441c22e4864d797d7fc603dbec603a, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40715,1676264030105, regionLocation=jenkins-hbase4.apache.org,40715,1676264030105, openSeqNum=2 2023-02-13 04:53:57,565 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=98f1b8d5338c83a011c1a33d5b920f87, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40473,1676264030129, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129, openSeqNum=2 2023-02-13 04:53:57,566 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=bd5cf65a2c7621de28620d9e1b36b8d7, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40473,1676264030129, regionLocation=jenkins-hbase4.apache.org,40473,1676264030129, openSeqNum=2 2023-02-13 04:53:57,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 04:53:57,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676264097566 2023-02-13 04:53:57,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676264157566 2023-02-13 04:53:57,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 8 msec 2023-02-13 04:53:57,588 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,34837,1676264030153 had 1 regions 2023-02-13 04:53:57,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33465,1676264036173-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:57,588 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,40473,1676264030129 had 3 regions 2023-02-13 04:53:57,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33465,1676264036173-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:57,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33465,1676264036173-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:57,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:33465, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:57,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 04:53:57,588 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,40715,1676264030105 had 1 regions 2023-02-13 04:53:57,588 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. is NOT online; state={b37bc85591cde75b9b23b10dea6af810 state=OPEN, ts=1676264037565, server=jenkins-hbase4.apache.org,40473,1676264030129}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 04:53:57,589 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,40473,1676264030129, splitWal=true, meta=false, isMeta: false 2023-02-13 04:53:57,589 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,40715,1676264030105, splitWal=true, meta=false, isMeta: false 2023-02-13 04:53:57,591 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34837,1676264030153, splitWal=true, meta=true, isMeta: false 2023-02-13 04:53:57,592 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129-splitting 2023-02-13 04:53:57,593 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129-splitting dir is empty, no logs to split. 2023-02-13 04:53:57,593 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,40473,1676264030129 WAL count=0, meta=false 2023-02-13 04:53:57,593 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105-splitting 2023-02-13 04:53:57,594 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105-splitting dir is empty, no logs to split. 2023-02-13 04:53:57,594 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,40715,1676264030105 WAL count=0, meta=false 2023-02-13 04:53:57,596 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,40473,1676264030129/hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810., unknown_server=jenkins-hbase4.apache.org,40715,1676264030105/restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a., unknown_server=jenkins-hbase4.apache.org,40473,1676264030129/restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87., unknown_server=jenkins-hbase4.apache.org,40473,1676264030129/restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:57,596 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153-splitting dir is empty, no logs to split. 2023-02-13 04:53:57,596 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,34837,1676264030153 WAL count=0, meta=false 2023-02-13 04:53:57,597 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129-splitting dir is empty, no logs to split. 2023-02-13 04:53:57,597 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,40473,1676264030129 WAL count=0, meta=false 2023-02-13 04:53:57,597 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,40473,1676264030129 WAL splitting is done? wals=0, meta=false 2023-02-13 04:53:57,598 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105-splitting dir is empty, no logs to split. 2023-02-13 04:53:57,598 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,40715,1676264030105 WAL count=0, meta=false 2023-02-13 04:53:57,598 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,40715,1676264030105 WAL splitting is done? wals=0, meta=false 2023-02-13 04:53:57,599 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,34837,1676264030153-splitting dir is empty, no logs to split. 2023-02-13 04:53:57,599 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,34837,1676264030153 WAL count=0, meta=false 2023-02-13 04:53:57,599 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,34837,1676264030153 WAL splitting is done? wals=0, meta=false 2023-02-13 04:53:57,599 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,40473,1676264030129 failed, ignore...File hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40473,1676264030129-splitting does not exist. 2023-02-13 04:53:57,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN}, {pid=24, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN}, {pid=25, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN}] 2023-02-13 04:53:57,601 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,40715,1676264030105 failed, ignore...File hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/WALs/jenkins-hbase4.apache.org,40715,1676264030105-splitting does not exist. 2023-02-13 04:53:57,601 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,34837,1676264030153 after splitting done 2023-02-13 04:53:57,601 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,34837,1676264030153 from processing; numProcessing=2 2023-02-13 04:53:57,602 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN 2023-02-13 04:53:57,602 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN}] 2023-02-13 04:53:57,602 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN 2023-02-13 04:53:57,602 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN 2023-02-13 04:53:57,603 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:57,603 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,40473,1676264030129; forceNewPlan=false, retain=true 2023-02-13 04:53:57,603 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:57,603 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40473,1676264030129 2023-02-13 04:53:57,603 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,40473,1676264030129; forceNewPlan=false, retain=true 2023-02-13 04:53:57,603 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN 2023-02-13 04:53:57,603 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,40473,1676264030129; forceNewPlan=false, retain=true 2023-02-13 04:53:57,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,34837,1676264030153, splitWal=true, meta=true in 1.2740 sec 2023-02-13 04:53:57,604 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40715,1676264030105 2023-02-13 04:53:57,604 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,40715,1676264030105; forceNewPlan=false, retain=true 2023-02-13 04:53:57,753 INFO [jenkins-hbase4:33465] balancer.BaseLoadBalancer(1545): Reassigned 4 regions. 4 retained the pre-restart assignment. 2023-02-13 04:53:57,756 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b37bc85591cde75b9b23b10dea6af810, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,756 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=98f1b8d5338c83a011c1a33d5b920f87, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,756 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=f0441c22e4864d797d7fc603dbec603a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:57,756 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264037756"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264037756"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264037756"}]},"ts":"1676264037756"} 2023-02-13 04:53:57,756 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=bd5cf65a2c7621de28620d9e1b36b8d7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,756 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264037756"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264037756"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264037756"}]},"ts":"1676264037756"} 2023-02-13 04:53:57,756 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264037756"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264037756"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264037756"}]},"ts":"1676264037756"} 2023-02-13 04:53:57,756 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264037756"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264037756"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264037756"}]},"ts":"1676264037756"} 2023-02-13 04:53:57,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure 98f1b8d5338c83a011c1a33d5b920f87, server=jenkins-hbase4.apache.org,40473,1676264036416}] 2023-02-13 04:53:57,760 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=26, state=RUNNABLE; OpenRegionProcedure f0441c22e4864d797d7fc603dbec603a, server=jenkins-hbase4.apache.org,40715,1676264036203}] 2023-02-13 04:53:57,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; OpenRegionProcedure bd5cf65a2c7621de28620d9e1b36b8d7, server=jenkins-hbase4.apache.org,40473,1676264036416}] 2023-02-13 04:53:57,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; OpenRegionProcedure b37bc85591cde75b9b23b10dea6af810, server=jenkins-hbase4.apache.org,40473,1676264036416}] 2023-02-13 04:53:57,913 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,913 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 04:53:57,915 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45438, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 04:53:57,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:57,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(7850): Opening region: {ENCODED => f0441c22e4864d797d7fc603dbec603a, NAME => 'restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:57,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:57,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(7890): checking encryption for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(7893): checking classloading for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,921 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:57,921 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7850): Opening region: {ENCODED => b37bc85591cde75b9b23b10dea6af810, NAME => 'hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:57,921 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,922 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:57,922 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7890): checking encryption for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,922 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7893): checking classloading for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,929 INFO [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:57,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7850): Opening region: {ENCODED => 98f1b8d5338c83a011c1a33d5b920f87, NAME => 'restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:57,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:57,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7890): checking encryption for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7893): checking classloading for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,933 DEBUG [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/family 2023-02-13 04:53:57,933 DEBUG [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/family 2023-02-13 04:53:57,933 INFO [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f0441c22e4864d797d7fc603dbec603a columnFamilyName family 2023-02-13 04:53:57,934 INFO [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,935 INFO [StoreOpener-f0441c22e4864d797d7fc603dbec603a-1] regionserver.HStore(310): Store=f0441c22e4864d797d7fc603dbec603a/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:57,936 INFO [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,937 DEBUG [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info 2023-02-13 04:53:57,937 DEBUG [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info 2023-02-13 04:53:57,937 INFO [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b37bc85591cde75b9b23b10dea6af810 columnFamilyName info 2023-02-13 04:53:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,938 DEBUG [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/family 2023-02-13 04:53:57,938 DEBUG [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/family 2023-02-13 04:53:57,939 INFO [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98f1b8d5338c83a011c1a33d5b920f87 columnFamilyName family 2023-02-13 04:53:57,940 INFO [StoreOpener-98f1b8d5338c83a011c1a33d5b920f87-1] regionserver.HStore(310): Store=98f1b8d5338c83a011c1a33d5b920f87/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:57,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1054): writing seq id for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:57,947 DEBUG [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] regionserver.HStore(539): loaded hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/info/c470f27a0d0948e99e0550e711f95a3e 2023-02-13 04:53:57,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1071): Opened f0441c22e4864d797d7fc603dbec603a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59636002, jitterRate=-0.11135432124137878}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:57,948 INFO [StoreOpener-b37bc85591cde75b9b23b10dea6af810-1] regionserver.HStore(310): Store=b37bc85591cde75b9b23b10dea6af810/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:57,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(964): Region open journal for f0441c22e4864d797d7fc603dbec603a: Running coprocessor pre-open hook at 1676264037920Writing region info on filesystem at 1676264037920Initializing all the Stores at 1676264037921 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264037922 (+1 ms)Cleaning up temporary data from old regions at 1676264037941 (+19 ms)Cleaning up detritus from prior splits at 1676264037942 (+1 ms)Running coprocessor post-open hooks at 1676264037948 (+6 ms)Region opened successfully at 1676264037948 2023-02-13 04:53:57,949 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a., pid=28, masterSystemTime=1676264037913 2023-02-13 04:53:57,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1054): writing seq id for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:57,952 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:57,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40715-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:57,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1071): Opened 98f1b8d5338c83a011c1a33d5b920f87; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65411978, jitterRate=-0.025285571813583374}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:57,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(964): Region open journal for 98f1b8d5338c83a011c1a33d5b920f87: Running coprocessor pre-open hook at 1676264037933Writing region info on filesystem at 1676264037933Initializing all the Stores at 1676264037935 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264037935Cleaning up temporary data from old regions at 1676264037945 (+10 ms)Cleaning up detritus from prior splits at 1676264037945Running coprocessor post-open hooks at 1676264037955 (+10 ms)Region opened successfully at 1676264037955 2023-02-13 04:53:57,955 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=f0441c22e4864d797d7fc603dbec603a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:57,956 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264037955"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264037955"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264037955"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264037955"}]},"ts":"1676264037955"} 2023-02-13 04:53:57,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87., pid=27, masterSystemTime=1676264037913 2023-02-13 04:53:57,962 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1054): writing seq id for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:57,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:57,966 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1071): Opened b37bc85591cde75b9b23b10dea6af810; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59598872, jitterRate=-0.11190760135650635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:57,966 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(964): Region open journal for b37bc85591cde75b9b23b10dea6af810: Running coprocessor pre-open hook at 1676264037922Writing region info on filesystem at 1676264037922Initializing all the Stores at 1676264037933 (+11 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264037933Cleaning up temporary data from old regions at 1676264037954 (+21 ms)Cleaning up detritus from prior splits at 1676264037956 (+2 ms)Running coprocessor post-open hooks at 1676264037966 (+10 ms)Region opened successfully at 1676264037966 2023-02-13 04:53:57,966 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=98f1b8d5338c83a011c1a33d5b920f87, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,966 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676264037966"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264037966"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264037966"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264037966"}]},"ts":"1676264037966"} 2023-02-13 04:53:57,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:57,968 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810., pid=30, masterSystemTime=1676264037913 2023-02-13 04:53:57,967 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=26 2023-02-13 04:53:57,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:57,968 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=26, state=SUCCESS; OpenRegionProcedure f0441c22e4864d797d7fc603dbec603a, server=jenkins-hbase4.apache.org,40715,1676264036203 in 199 msec 2023-02-13 04:53:57,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd5cf65a2c7621de28620d9e1b36b8d7, NAME => 'restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.', STARTKEY => '', ENDKEY => ''} 2023-02-13 04:53:57,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 04:53:57,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7890): checking encryption for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(7893): checking classloading for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,970 INFO [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,971 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-02-13 04:53:57,971 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,40715,1676264030105 after splitting done 2023-02-13 04:53:57,971 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b37bc85591cde75b9b23b10dea6af810, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,972 DEBUG [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/family 2023-02-13 04:53:57,971 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=f0441c22e4864d797d7fc603dbec603a, ASSIGN in 366 msec 2023-02-13 04:53:57,972 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:57,972 DEBUG [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/family 2023-02-13 04:53:57,972 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264037971"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264037971"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264037971"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264037971"}]},"ts":"1676264037971"} 2023-02-13 04:53:57,971 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,40715,1676264030105 from processing; numProcessing=1 2023-02-13 04:53:57,972 INFO [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd5cf65a2c7621de28620d9e1b36b8d7 columnFamilyName family 2023-02-13 04:53:57,974 INFO [StoreOpener-bd5cf65a2c7621de28620d9e1b36b8d7-1] regionserver.HStore(310): Store=bd5cf65a2c7621de28620d9e1b36b8d7/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 04:53:57,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-02-13 04:53:57,976 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure 98f1b8d5338c83a011c1a33d5b920f87, server=jenkins-hbase4.apache.org,40473,1676264036416 in 211 msec 2023-02-13 04:53:57,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,40715,1676264030105, splitWal=true, meta=false in 1.6470 sec 2023-02-13 04:53:57,972 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:40473-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:57,977 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-13 04:53:57,977 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=98f1b8d5338c83a011c1a33d5b920f87, ASSIGN in 376 msec 2023-02-13 04:53:57,977 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; OpenRegionProcedure b37bc85591cde75b9b23b10dea6af810, server=jenkins-hbase4.apache.org,40473,1676264036416 in 210 msec 2023-02-13 04:53:57,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b37bc85591cde75b9b23b10dea6af810, ASSIGN in 377 msec 2023-02-13 04:53:57,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1054): writing seq id for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:57,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1071): Opened bd5cf65a2c7621de28620d9e1b36b8d7; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74527321, jitterRate=0.1105436235666275}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 04:53:57,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(964): Region open journal for bd5cf65a2c7621de28620d9e1b36b8d7: Running coprocessor pre-open hook at 1676264037969Writing region info on filesystem at 1676264037969Initializing all the Stores at 1676264037970 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264037970Cleaning up temporary data from old regions at 1676264037981 (+11 ms)Cleaning up detritus from prior splits at 1676264037981Running coprocessor post-open hooks at 1676264037985 (+4 ms)Region opened successfully at 1676264037985 2023-02-13 04:53:57,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7., pid=29, masterSystemTime=1676264037913 2023-02-13 04:53:57,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:57,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:40473-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:57,988 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=bd5cf65a2c7621de28620d9e1b36b8d7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:57,988 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676264037988"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264037988"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264037988"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264037988"}]},"ts":"1676264037988"} 2023-02-13 04:53:57,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-13 04:53:57,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; OpenRegionProcedure bd5cf65a2c7621de28620d9e1b36b8d7, server=jenkins-hbase4.apache.org,40473,1676264036416 in 229 msec 2023-02-13 04:53:57,996 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=18 2023-02-13 04:53:57,996 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,40473,1676264030129 after splitting done 2023-02-13 04:53:57,996 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=bd5cf65a2c7621de28620d9e1b36b8d7, ASSIGN in 392 msec 2023-02-13 04:53:57,996 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,40473,1676264030129 from processing; numProcessing=0 2023-02-13 04:53:57,999 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,40473,1676264030129, splitWal=true, meta=false in 1.6740 sec 2023-02-13 04:53:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 04:53:58,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33465] ipc.CallRunner(144): callId: 58 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:59748 deadline: 1676264098042, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 04:53:58,589 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 04:53:58,594 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:58,595 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45446, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-13 04:53:58,608 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 04:53:58,611 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 04:53:58,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.415sec 2023-02-13 04:53:58,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 04:53:58,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 04:53:58,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 04:53:58,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33465,1676264036173-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 04:53:58,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33465,1676264036173-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 04:53:58,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 04:53:58,696 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 04:53:58,902 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 04:53:58,903 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 04:53:59,052 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:59,059 DEBUG [hconnection-0x109ae3a3-shared-pool-9] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 04:53:59,061 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45458, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 04:53:59,061 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:59,068 INFO [Listener at localhost/33627] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 04:53:59,068 INFO [Listener at localhost/33627] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 04:53:59,072 INFO [Listener at localhost/33627] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 04:53:59,073 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => bd5cf65a2c7621de28620d9e1b36b8d7, NAME => 'restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,40473,1676264030129, currentServer=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:59,073 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 98f1b8d5338c83a011c1a33d5b920f87, NAME => 'restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,40473,1676264030129, currentServer=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:59,073 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => b37bc85591cde75b9b23b10dea6af810, NAME => 'hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,40473,1676264030129, currentServer=jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:59,073 INFO [Listener at localhost/33627] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => f0441c22e4864d797d7fc603dbec603a, NAME => 'restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,40715,1676264030105, currentServer=jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:59,073 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 04:53:59,073 INFO [Listener at localhost/33627] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 04:53:59,073 DEBUG [Listener at localhost/33627] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7dffd1a0 to 127.0.0.1:59910 2023-02-13 04:53:59,073 DEBUG [Listener at localhost/33627] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,074 DEBUG [Listener at localhost/33627] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 04:53:59,074 DEBUG [Listener at localhost/33627] util.JVMClusterUtil(257): Found active master hash=1029449637, stopped=false 2023-02-13 04:53:59,074 INFO [Listener at localhost/33627] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:59,076 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:59,076 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:59,077 INFO [Listener at localhost/33627] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 04:53:59,076 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:59,076 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 04:53:59,077 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:59,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:59,079 DEBUG [Listener at localhost/33627] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7785a270 to 127.0.0.1:59910 2023-02-13 04:53:59,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:59,079 DEBUG [Listener at localhost/33627] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:59,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 04:53:59,080 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40715,1676264036203' ***** 2023-02-13 04:53:59,080 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:59,080 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40473,1676264036416' ***** 2023-02-13 04:53:59,080 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:59,080 INFO [Listener at localhost/33627] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34837,1676264036498' ***** 2023-02-13 04:53:59,080 INFO [RS:4;jenkins-hbase4:40473] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:59,080 INFO [Listener at localhost/33627] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 04:53:59,080 INFO [RS:3;jenkins-hbase4:40715] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:59,081 INFO [RS:5;jenkins-hbase4:34837] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 04:53:59,081 INFO [RS:4;jenkins-hbase4:40473] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:59,081 INFO [RS:5;jenkins-hbase4:34837] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:59,081 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:59,081 INFO [RS:5;jenkins-hbase4:34837] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:59,081 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:59,081 INFO [RS:4;jenkins-hbase4:40473] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:59,081 INFO [RS:3;jenkins-hbase4:40715] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 04:53:59,081 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 04:53:59,082 INFO [RS:3;jenkins-hbase4:40715] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 04:53:59,082 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(3304): Received CLOSE for b37bc85591cde75b9b23b10dea6af810 2023-02-13 04:53:59,083 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(3304): Received CLOSE for f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:59,081 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:59,083 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(3304): Received CLOSE for 98f1b8d5338c83a011c1a33d5b920f87 2023-02-13 04:53:59,083 DEBUG [RS:5;jenkins-hbase4:34837] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5acb77fc to 127.0.0.1:59910 2023-02-13 04:53:59,083 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:59,083 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(3304): Received CLOSE for bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:59,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1603): Closing f0441c22e4864d797d7fc603dbec603a, disabling compactions & flushes 2023-02-13 04:53:59,084 DEBUG [RS:3;jenkins-hbase4:40715] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x665c3c71 to 127.0.0.1:59910 2023-02-13 04:53:59,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1603): Closing b37bc85591cde75b9b23b10dea6af810, disabling compactions & flushes 2023-02-13 04:53:59,083 DEBUG [RS:5;jenkins-hbase4:34837] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,084 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34837,1676264036498; all regions closed. 2023-02-13 04:53:59,084 DEBUG [RS:3;jenkins-hbase4:40715] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:59,084 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:59,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:59,085 DEBUG [RS:4;jenkins-hbase4:40473] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01af5c95 to 127.0.0.1:59910 2023-02-13 04:53:59,085 DEBUG [RS:4;jenkins-hbase4:40473] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:59,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. after waiting 0 ms 2023-02-13 04:53:59,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:59,084 INFO [RS:3;jenkins-hbase4:40715] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:59,085 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 04:53:59,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:59,085 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1479): Online Regions={b37bc85591cde75b9b23b10dea6af810=hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810., 98f1b8d5338c83a011c1a33d5b920f87=restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87., bd5cf65a2c7621de28620d9e1b36b8d7=restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7.} 2023-02-13 04:53:59,085 INFO [RS:3;jenkins-hbase4:40715] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:59,085 DEBUG [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1505): Waiting on 98f1b8d5338c83a011c1a33d5b920f87, b37bc85591cde75b9b23b10dea6af810, bd5cf65a2c7621de28620d9e1b36b8d7 2023-02-13 04:53:59,085 INFO [RS:3;jenkins-hbase4:40715] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:59,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. after waiting 0 ms 2023-02-13 04:53:59,085 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 04:53:59,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:59,088 INFO [regionserver/jenkins-hbase4:40715.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:59,088 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 04:53:59,088 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1479): Online Regions={f0441c22e4864d797d7fc603dbec603a=restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 04:53:59,088 DEBUG [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1505): Waiting on 1588230740, f0441c22e4864d797d7fc603dbec603a 2023-02-13 04:53:59,089 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 04:53:59,089 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 04:53:59,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 04:53:59,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 04:53:59,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 04:53:59,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=3.81 KB heapSize=6.91 KB 2023-02-13 04:53:59,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableOne/f0441c22e4864d797d7fc603dbec603a/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 04:53:59,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1837): Closed restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:59,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1557): Region close journal for f0441c22e4864d797d7fc603dbec603a: Waiting for close lock at 1676264039084Running coprocessor pre-close hooks at 1676264039084Disabling compacts and flushes for region at 1676264039084Disabling writes for close at 1676264039085 (+1 ms)Writing region close event to WAL at 1676264039086 (+1 ms)Running coprocessor post-close hooks at 1676264039096 (+10 ms)Closed at 1676264039096 2023-02-13 04:53:59,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40715-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676264031593.f0441c22e4864d797d7fc603dbec603a. 2023-02-13 04:53:59,102 DEBUG [RS:5;jenkins-hbase4:34837] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:59,102 INFO [RS:5;jenkins-hbase4:34837] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34837%2C1676264036498:(num 1676264036704) 2023-02-13 04:53:59,102 DEBUG [RS:5;jenkins-hbase4:34837] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,102 INFO [RS:5;jenkins-hbase4:34837] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:59,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/namespace/b37bc85591cde75b9b23b10dea6af810/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-02-13 04:53:59,107 INFO [RS:5;jenkins-hbase4:34837] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:34837 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:59,108 INFO [RS:5;jenkins-hbase4:34837] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:59,108 INFO [regionserver/jenkins-hbase4:34837.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:59,108 INFO [RS:5;jenkins-hbase4:34837] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:59,108 INFO [RS:5;jenkins-hbase4:34837] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:59,109 INFO [RS:5;jenkins-hbase4:34837] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34837 2023-02-13 04:53:59,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:59,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1557): Region close journal for b37bc85591cde75b9b23b10dea6af810: Waiting for close lock at 1676264039083Running coprocessor pre-close hooks at 1676264039083Disabling compacts and flushes for region at 1676264039083Disabling writes for close at 1676264039086 (+3 ms)Writing region close event to WAL at 1676264039092 (+6 ms)Running coprocessor post-close hooks at 1676264039111 (+19 ms)Closed at 1676264039111 2023-02-13 04:53:59,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676264030929.b37bc85591cde75b9b23b10dea6af810. 2023-02-13 04:53:59,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1603): Closing 98f1b8d5338c83a011c1a33d5b920f87, disabling compactions & flushes 2023-02-13 04:53:59,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:59,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:59,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. after waiting 0 ms 2023-02-13 04:53:59,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:59,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableThree/98f1b8d5338c83a011c1a33d5b920f87/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 04:53:59,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1837): Closed restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:59,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1557): Region close journal for 98f1b8d5338c83a011c1a33d5b920f87: Waiting for close lock at 1676264039112Running coprocessor pre-close hooks at 1676264039112Disabling compacts and flushes for region at 1676264039112Disabling writes for close at 1676264039112Writing region close event to WAL at 1676264039113 (+1 ms)Running coprocessor post-close hooks at 1676264039126 (+13 ms)Closed at 1676264039127 (+1 ms) 2023-02-13 04:53:59,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676264032824.98f1b8d5338c83a011c1a33d5b920f87. 2023-02-13 04:53:59,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1603): Closing bd5cf65a2c7621de28620d9e1b36b8d7, disabling compactions & flushes 2023-02-13 04:53:59,128 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:59,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:59,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. after waiting 0 ms 2023-02-13 04:53:59,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:59,130 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.81 KB at sequenceid=38 (bloomFilter=false), to=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/.tmp/info/c676f01f37bf4470bc939ac6bde2f27e 2023-02-13 04:53:59,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/default/restartTableTwo/bd5cf65a2c7621de28620d9e1b36b8d7/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 04:53:59,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:59,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] regionserver.HRegion(1557): Region close journal for bd5cf65a2c7621de28620d9e1b36b8d7: Waiting for close lock at 1676264039127Running coprocessor pre-close hooks at 1676264039127Disabling compacts and flushes for region at 1676264039127Disabling writes for close at 1676264039128 (+1 ms)Writing region close event to WAL at 1676264039130 (+2 ms)Running coprocessor post-close hooks at 1676264039135 (+5 ms)Closed at 1676264039135 2023-02-13 04:53:59,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:40473-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676264032209.bd5cf65a2c7621de28620d9e1b36b8d7. 2023-02-13 04:53:59,142 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/.tmp/info/c676f01f37bf4470bc939ac6bde2f27e as hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info/c676f01f37bf4470bc939ac6bde2f27e 2023-02-13 04:53:59,149 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HStore(1080): Added hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/info/c676f01f37bf4470bc939ac6bde2f27e, entries=32, sequenceid=38, filesize=8.5 K 2023-02-13 04:53:59,152 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(2947): Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 61ms, sequenceid=38, compaction requested=false 2023-02-13 04:53:59,167 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/data/hbase/meta/1588230740/recovered.edits/41.seqid, newMaxSeqId=41, maxSeqId=27 2023-02-13 04:53:59,168 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 04:53:59,169 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:59,169 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264039088Running coprocessor pre-close hooks at 1676264039088Disabling compacts and flushes for region at 1676264039088Disabling writes for close at 1676264039090 (+2 ms)Obtaining lock to block concurrent updates at 1676264039090Preparing flush snapshotting stores in 1588230740 at 1676264039090Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=3900, getHeapSize=7032, getOffHeapSize=0, getCellsCount=32 at 1676264039090Flushing stores of hbase:meta,,1.1588230740 at 1676264039092 (+2 ms)Flushing 1588230740/info: creating writer at 1676264039092Flushing 1588230740/info: appending metadata at 1676264039095 (+3 ms)Flushing 1588230740/info: closing flushed file at 1676264039096 (+1 ms)Flushing 1588230740/info: reopening flushed file at 1676264039142 (+46 ms)Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 61ms, sequenceid=38, compaction requested=false at 1676264039152 (+10 ms)Writing region close event to WAL at 1676264039162 (+10 ms)Running coprocessor post-close hooks at 1676264039168 (+6 ms)Closed at 1676264039169 (+1 ms) 2023-02-13 04:53:59,169 INFO [regionserver/jenkins-hbase4:40473.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:59,169 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:40715-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 04:53:59,170 INFO [regionserver/jenkins-hbase4:34837.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:59,192 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:59,192 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6f91bffa rejected from java.util.concurrent.ThreadPoolExecutor@6bae36d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:59,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:59,192 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34837,1676264036498 2023-02-13 04:53:59,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:59,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:59,193 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:59,193 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d4456e9 rejected from java.util.concurrent.ThreadPoolExecutor@6bae36d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,195 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34837,1676264036498] 2023-02-13 04:53:59,195 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34837,1676264036498; numProcessing=1 2023-02-13 04:53:59,196 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34837,1676264036498 already deleted, retry=false 2023-02-13 04:53:59,196 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34837,1676264036498 expired; onlineServers=2 2023-02-13 04:53:59,285 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40473,1676264036416; all regions closed. 2023-02-13 04:53:59,288 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40715,1676264036203; all regions closed. 2023-02-13 04:53:59,295 DEBUG [RS:4;jenkins-hbase4:40473] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:59,295 INFO [RS:4;jenkins-hbase4:40473] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40473%2C1676264036416:(num 1676264036594) 2023-02-13 04:53:59,295 DEBUG [RS:4;jenkins-hbase4:40473] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,295 INFO [RS:4;jenkins-hbase4:40473] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:59,296 INFO [RS:4;jenkins-hbase4:40473] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:40473 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:59,296 INFO [RS:4;jenkins-hbase4:40473] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 04:53:59,296 INFO [regionserver/jenkins-hbase4:40473.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:59,296 INFO [RS:4;jenkins-hbase4:40473] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 04:53:59,296 INFO [RS:4;jenkins-hbase4:40473] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 04:53:59,297 INFO [RS:4;jenkins-hbase4:40473] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40473 2023-02-13 04:53:59,299 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:59,299 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40473,1676264036416 2023-02-13 04:53:59,299 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:59,300 DEBUG [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:59,300 INFO [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40715%2C1676264036203.meta:.meta(num 1676264036700) 2023-02-13 04:53:59,302 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40473,1676264036416] 2023-02-13 04:53:59,302 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40473,1676264036416; numProcessing=2 2023-02-13 04:53:59,303 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40473,1676264036416 already deleted, retry=false 2023-02-13 04:53:59,303 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40473,1676264036416 expired; onlineServers=1 2023-02-13 04:53:59,310 DEBUG [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/oldWALs 2023-02-13 04:53:59,310 INFO [RS:3;jenkins-hbase4:40715] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40715%2C1676264036203:(num 1676264036511) 2023-02-13 04:53:59,310 DEBUG [RS:3;jenkins-hbase4:40715] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,310 INFO [RS:3;jenkins-hbase4:40715] regionserver.LeaseManager(133): Closed leases 2023-02-13 04:53:59,310 INFO [RS:3;jenkins-hbase4:40715] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:40715 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 04:53:59,310 INFO [regionserver/jenkins-hbase4:40715.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:59,311 INFO [RS:3;jenkins-hbase4:40715] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40715 2023-02-13 04:53:59,313 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40715,1676264036203 2023-02-13 04:53:59,313 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 04:53:59,313 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6cbeaafa rejected from java.util.concurrent.ThreadPoolExecutor@688123b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,314 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40715,1676264036203] 2023-02-13 04:53:59,314 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40715,1676264036203; numProcessing=3 2023-02-13 04:53:59,316 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40715,1676264036203 already deleted, retry=false 2023-02-13 04:53:59,316 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40715,1676264036203 expired; onlineServers=0 2023-02-13 04:53:59,316 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33465,1676264036173' ***** 2023-02-13 04:53:59,316 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 04:53:59,317 DEBUG [M:1;jenkins-hbase4:33465] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3963b4d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 04:53:59,317 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33465,1676264036173 2023-02-13 04:53:59,317 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33465,1676264036173; all regions closed. 2023-02-13 04:53:59,317 DEBUG [M:1;jenkins-hbase4:33465] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 04:53:59,317 DEBUG [M:1;jenkins-hbase4:33465] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 04:53:59,317 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 04:53:59,317 DEBUG [M:1;jenkins-hbase4:33465] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 04:53:59,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264036353] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264036353,5,FailOnTimeoutGroup] 2023-02-13 04:53:59,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264036353] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264036353,5,FailOnTimeoutGroup] 2023-02-13 04:53:59,317 INFO [M:1;jenkins-hbase4:33465] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 04:53:59,318 INFO [M:1;jenkins-hbase4:33465] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 04:53:59,318 INFO [M:1;jenkins-hbase4:33465] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 04:53:59,318 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 04:53:59,319 DEBUG [M:1;jenkins-hbase4:33465] master.HMaster(1502): Stopping service threads 2023-02-13 04:53:59,319 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 04:53:59,319 INFO [M:1;jenkins-hbase4:33465] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 04:53:59,319 ERROR [M:1;jenkins-hbase4:33465] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 04:53:59,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 04:53:59,319 INFO [M:1;jenkins-hbase4:33465] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 04:53:59,319 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 04:53:59,320 DEBUG [M:1;jenkins-hbase4:33465] zookeeper.ZKUtil(398): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 04:53:59,320 WARN [M:1;jenkins-hbase4:33465] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 04:53:59,320 INFO [M:1;jenkins-hbase4:33465] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 04:53:59,320 INFO [M:1;jenkins-hbase4:33465] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 04:53:59,321 DEBUG [M:1;jenkins-hbase4:33465] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 04:53:59,321 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:59,321 DEBUG [M:1;jenkins-hbase4:33465] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:59,321 DEBUG [M:1;jenkins-hbase4:33465] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 04:53:59,321 DEBUG [M:1;jenkins-hbase4:33465] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:59,321 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.73 KB heapSize=47.93 KB 2023-02-13 04:53:59,339 INFO [M:1;jenkins-hbase4:33465] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.73 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/700df85c3ece40c6bc219654c8cf15d2 2023-02-13 04:53:59,344 DEBUG [M:1;jenkins-hbase4:33465] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/700df85c3ece40c6bc219654c8cf15d2 as hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/700df85c3ece40c6bc219654c8cf15d2 2023-02-13 04:53:59,350 INFO [M:1;jenkins-hbase4:33465] regionserver.HStore(1080): Added hdfs://localhost:44887/user/jenkins/test-data/4c320bf6-b5d9-336e-b74f-491fb427c2ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/700df85c3ece40c6bc219654c8cf15d2, entries=13, sequenceid=246, filesize=6.2 K 2023-02-13 04:53:59,351 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegion(2947): Finished flush of dataSize ~38.73 KB/39655, heapSize ~47.91 KB/49064, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=246, compaction requested=false 2023-02-13 04:53:59,354 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 04:53:59,354 DEBUG [M:1;jenkins-hbase4:33465] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264039321Disabling compacts and flushes for region at 1676264039321Disabling writes for close at 1676264039321Obtaining lock to block concurrent updates at 1676264039321Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676264039321Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=39655, getHeapSize=49064, getOffHeapSize=0, getCellsCount=121 at 1676264039321Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676264039322 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676264039323 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676264039327 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676264039327Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676264039345 (+18 ms)Finished flush of dataSize ~38.73 KB/39655, heapSize ~47.91 KB/49064, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=246, compaction requested=false at 1676264039351 (+6 ms)Writing region close event to WAL at 1676264039354 (+3 ms)Closed at 1676264039354 2023-02-13 04:53:59,359 INFO [M:1;jenkins-hbase4:33465] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 04:53:59,360 INFO [M:1;jenkins-hbase4:33465] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33465 2023-02-13 04:53:59,360 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 04:53:59,362 DEBUG [M:1;jenkins-hbase4:33465] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,33465,1676264036173 already deleted, retry=false 2023-02-13 04:53:59,577 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,577 INFO [M:1;jenkins-hbase4:33465] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33465,1676264036173; zookeeper connection closed. 2023-02-13 04:53:59,577 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@67bc6686 rejected from java.util.concurrent.ThreadPoolExecutor@6383eea9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,578 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): master:33465-0x1004c0729060009, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,578 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@13e170f9 rejected from java.util.concurrent.ThreadPoolExecutor@6383eea9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,677 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,677 INFO [RS:3;jenkins-hbase4:40715] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40715,1676264036203; zookeeper connection closed. 2023-02-13 04:53:59,677 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@594fad4 rejected from java.util.concurrent.ThreadPoolExecutor@688123b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,678 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40715-0x1004c072906000a, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,678 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1f8c8365] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1f8c8365 2023-02-13 04:53:59,678 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d196d44 rejected from java.util.concurrent.ThreadPoolExecutor@688123b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,777 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,777 INFO [RS:4;jenkins-hbase4:40473] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40473,1676264036416; zookeeper connection closed. 2023-02-13 04:53:59,778 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4081bff4 rejected from java.util.concurrent.ThreadPoolExecutor@44be333c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,778 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7a6b588c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7a6b588c 2023-02-13 04:53:59,778 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:40473-0x1004c072906000d, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,778 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7dc4098c rejected from java.util.concurrent.ThreadPoolExecutor@44be333c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,878 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,878 INFO [RS:5;jenkins-hbase4:34837] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34837,1676264036498; zookeeper connection closed. 2023-02-13 04:53:59,878 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5cae54b rejected from java.util.concurrent.ThreadPoolExecutor@6bae36d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,878 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4dc23df] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4dc23df 2023-02-13 04:53:59,878 DEBUG [Listener at localhost/33627-EventThread] zookeeper.ZKWatcher(600): regionserver:34837-0x1004c072906000f, quorum=127.0.0.1:59910, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 04:53:59,878 ERROR [Listener at localhost/33627-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@112fbea9 rejected from java.util.concurrent.ThreadPoolExecutor@6bae36d0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 04:53:59,878 INFO [Listener at localhost/33627] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 6 regionserver(s) complete 2023-02-13 04:53:59,879 WARN [Listener at localhost/33627] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 04:53:59,883 INFO [Listener at localhost/33627] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:53:59,987 WARN [BP-793984720-172.31.14.131-1676264029226 heartbeating to localhost/127.0.0.1:44887] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 04:53:59,987 WARN [BP-793984720-172.31.14.131-1676264029226 heartbeating to localhost/127.0.0.1:44887] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-793984720-172.31.14.131-1676264029226 (Datanode Uuid 709dad92-a625-4c99-8c68-4c7530104616) service to localhost/127.0.0.1:44887 2023-02-13 04:53:59,988 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/dfs/data/data5/current/BP-793984720-172.31.14.131-1676264029226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:59,988 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/dfs/data/data6/current/BP-793984720-172.31.14.131-1676264029226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:53:59,989 WARN [Listener at localhost/33627] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 04:53:59,993 INFO [Listener at localhost/33627] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:54:00,099 WARN [BP-793984720-172.31.14.131-1676264029226 heartbeating to localhost/127.0.0.1:44887] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 04:54:00,099 WARN [BP-793984720-172.31.14.131-1676264029226 heartbeating to localhost/127.0.0.1:44887] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-793984720-172.31.14.131-1676264029226 (Datanode Uuid 3da01103-3cef-49b3-8ff9-04fda312a49c) service to localhost/127.0.0.1:44887 2023-02-13 04:54:00,099 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/dfs/data/data3/current/BP-793984720-172.31.14.131-1676264029226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:54:00,100 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/dfs/data/data4/current/BP-793984720-172.31.14.131-1676264029226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:54:00,101 WARN [Listener at localhost/33627] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 04:54:00,104 INFO [Listener at localhost/33627] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:54:00,208 WARN [BP-793984720-172.31.14.131-1676264029226 heartbeating to localhost/127.0.0.1:44887] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 04:54:00,208 WARN [BP-793984720-172.31.14.131-1676264029226 heartbeating to localhost/127.0.0.1:44887] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-793984720-172.31.14.131-1676264029226 (Datanode Uuid 1010f7f1-03e6-4a21-9a30-9dde67a8be02) service to localhost/127.0.0.1:44887 2023-02-13 04:54:00,209 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/dfs/data/data1/current/BP-793984720-172.31.14.131-1676264029226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:54:00,209 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/792f905b-e817-3924-a7de-d18feef7e991/cluster_63c61889-be39-afbe-7dcf-02994a6c9fb7/dfs/data/data2/current/BP-793984720-172.31.14.131-1676264029226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 04:54:00,220 INFO [Listener at localhost/33627] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 04:54:00,337 INFO [Listener at localhost/33627] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 04:54:00,372 INFO [Listener at localhost/33627] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 04:54:00,388 INFO [Listener at localhost/33627] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=156 (was 101) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/33627 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Parameter Sending Thread #3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:40473.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:40715.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:34837.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-13-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (1932900768) connection to localhost/127.0.0.1:44887 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:44887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=614 (was 504) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=279 (was 320), ProcessCount=168 (was 168), AvailableMemoryMB=3991 (was 4218)